Target identification and positioning method and system based on panoramic camera and laser radar

Abstract

The invention discloses a target identification and positioning method and system based on a panorama camera and a laser radar, and the method comprises the steps: obtaining video streams collected by the panorama camera and the laser radar for a to-be-identified scene, and generating batch processing frames according to the video streams; performing target identification on the batch processing frame based on a target identification model to obtain target point cloud data corresponding to the same target in the laser radar video stream of each camera video stream in the panorama camera; for the cut target point cloud data corresponding to each path of camera, calculating pixel coordinates of the target point cloud data on the two-dimensional image based on a calibration result; all pixel coordinates of the target corresponding to each path of camera are clustered, and the geographic position of the current frame of each path of target is obtained; and repeatedly calculating to obtain a multi-frame geographic position of each path of target, and obtaining a final positioning result of the target. Therefore, the target positioning precision can be improved, and the problems of inaccurate patrol target positioning and poor real-time performance are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of target recognition and positioning technology, and in particular to a target recognition and positioning method and system based on a panoramic camera and lidar. Background Technology

[0002] Urban infrastructure inspection is a crucial aspect of urban infrastructure supervision. Inspecting infrastructure accurately identifies potential safety hazards, facilitating rapid response and reducing accidents. Measuring infrastructure extracts fundamental data, helping to clarify asset ownership and simplifying operation and management. Existing inspection methods primarily include manual and vehicle-mounted approaches. Manual methods are often labor-intensive, lack real-time performance, are costly, and have difficulty maintaining consistent inspection frequency. Vehicle-mounted inspection methods, on the other hand, utilize onboard inspection systems mounted on vehicles to acquire video data during transit. This data is then analyzed to identify and locate targets.

[0003] However, existing positioning technologies often employ a single sensor, such as visual positioning systems and lidar positioning systems. These methods are limited in accuracy due to inherent hardware limitations. For example, depth camera-based positioning methods are often limited to a range of only 20 meters due to the camera's depth measurement distance, and their depth measurement errors are relatively large, resulting in high positioning inaccuracies. While lidar-based positioning methods can obtain accurate depth information, they are often limited by the radar beamwidth, making it difficult to extract and effectively identify distant or small targets, leading to false positives and false negatives. Furthermore, current vehicle-mounted patrol systems are complex in structure and typically employ post-processing methods, resulting in low analysis efficiency and difficulty meeting the real-time requirements for target recognition and positioning. Summary of the Invention

[0004] This invention provides a target recognition and positioning method and system based on panoramic camera and lidar, which can improve target positioning accuracy, obtain the final true geographical location of the target, and effectively solve the problems of inaccurate target positioning and poor real-time performance during patrol surveys. Firstly, a target recognition and localization method based on a panoramic camera and LiDAR is provided, including: The panoramic camera and LiDAR are calibrated, and the video streams captured by the panoramic camera and LiDAR of the scene to be identified are obtained. Batch processing frames are generated based on the video streams. Based on the target recognition model, target recognition is performed on the batch processing frames to obtain the target point cloud data of the same target in the lidar video stream for each camera video stream in the panoramic camera; Based on the visible area parameters of each camera in the panoramic camera, the target point cloud data corresponding to each camera is cropped. For the target point cloud data corresponding to each camera after cropping, the pixel coordinates of the target point cloud data on the two-dimensional image are calculated based on the calibration results. Cluster all pixel coordinates of the target corresponding to each camera to obtain the geographical location of the target in the current frame; The multi-frame geographic location of each target is obtained by repeated calculation, and the final positioning result of the target is obtained based on the multi-frame geographic location of each target.

[0005] In some embodiments, the calibration of the panoramic camera and LiDAR includes: Acquire calibration board photos and corresponding radar point cloud data collected by each camera in the panoramic camera and the LiDAR against the calibration board; Based on the calibration board photos and the radar point cloud data, the intrinsic parameters, extrinsic parameters, and distortion parameters of each camera are obtained.

[0006] In some embodiments, generating batch frames from the video stream includes: The video stream is separated into video data and audio data using the nvurisrcbin component in the DeepStream software. The video data is used to generate batch frames based on the nvstreammux component in the DeepStream software.

[0007] In some embodiments, the step of performing target recognition on the batch of frames based on the target recognition model to obtain target point cloud data of the same target in the lidar video stream for each camera video stream in the panoramic camera includes: The YOLO-based target recognition model is converted from pt to onnx to generate the engine model; Based on the engine model, target recognition is performed on the batch processing frames, and the identified targets are marked with IDs to obtain target recognition frames. Based on the target recognition frames, target point cloud data of the same target in the lidar video stream of each camera video stream in the panoramic camera is obtained.

[0008] In some embodiments, cropping the target point cloud data corresponding to each camera based on the visible area parameters of each camera in the panoramic camera includes: Obtain the visible area parameters of each camera in the panoramic camera; Based on the visible area parameters of each camera, the point cloud region corresponding to each camera is obtained. The target point cloud data corresponding to each camera is cropped based on the point cloud region.

[0009] In some embodiments, calculating the pixel coordinates of the target point cloud data on a two-dimensional image based on the calibration results for the target point cloud data corresponding to each cropped camera includes: For the target point cloud data corresponding to each camera after cropping, the three-dimensional points of the target point cloud data are transformed from the world coordinate system to the camera coordinate system based on the extrinsic parameters of the calibration results. Projecting three-dimensional points in the camera coordinate system onto the normalized image plane yields two-dimensional coordinates. The two-dimensional coordinates are corrected based on the distortion parameters of the calibration results to obtain the corrected coordinates; The calibration coordinates are converted into pixel coordinates on a two-dimensional image based on the intrinsic parameters of the calibration results.

[0010] In some embodiments, the step of clustering all pixel coordinates of the target corresponding to each camera to obtain the geographical location of each target includes: For each camera, the pixel coordinates of the target are fused with the 3D point coordinates of the target point cloud data to obtain fused data; A rectangular recognition region is generated based on all pixel coordinates of the target, and a point cloud set within the rectangular recognition region is extracted from the fused data. A density-based spatial clustering algorithm is used to calculate the coordinates of the target center point with the highest density in the point cloud set using a weighted average method. Calculate the distance and angle of the target center point relative to the RTK device; The geographical location of each target is calculated based on the distance, the angle, and the latitude, longitude coordinates and heading angle of the RTK device.

[0011] In some embodiments, the method for calculating the geographical location of each target based on the distance, the angle, and the latitude, longitude coordinates and heading angle of the RTK device is shown in the following formula: ; Wherein, the direction angle is: ; The central angle corresponding to the distance between the target center point coordinates and the RTK device: ; The difference in longitude is: ; In the formula, and These represent the distance and angle of the target center point relative to the RTK device, respectively; R is the Earth's radius; ( ) represents the latitude and longitude coordinates of the RTK device; The heading angle of the RTK device; The target's geographical location; In order to put Convert to radians; To convert the direction angle α to radians.

[0012] In some embodiments, obtaining the final location result of the target based on the multi-frame geographic location of each target includes: The distance-weighted clustering algorithm clusters the geographic locations of each target across multiple frames to obtain the localization result for each target. The method for averaging the location results of all road targets to obtain the final location result of the target is shown in the following formula: ; In the formula, The final location result of the target; n is the number of cameras in the panoramic camera; This represents the location result of the target corresponding to the nth camera.

[0013] Secondly, a target recognition and positioning system based on a panoramic camera and LiDAR is provided, including: The calibration module is used to calibrate the panoramic camera and the LiDAR, acquire the video streams collected by the panoramic camera and the LiDAR of the scene to be identified, and generate batch processing frames based on the video streams. The target recognition module is communicatively connected to the calibration module and is used to perform target recognition on the batch processing frames based on the target recognition model to obtain the target point cloud data of each camera video stream in the panoramic camera for the same target in the lidar video stream. The cropping module is communicatively connected to the target recognition module and is used to crop the target point cloud data corresponding to each camera according to the visible area parameters of each camera in the panoramic camera. The coordinate transformation module is communicatively connected to the cropping module and is used to calculate the pixel coordinates of the target point cloud data on the two-dimensional image based on the calibration results for the target point cloud data corresponding to each cropped camera. A clustering module, communicatively connected to the coordinate transformation module, is used to cluster all pixel coordinates of the target corresponding to each camera to obtain the geographical location of each target; and, The final positioning module is communicatively connected to the clustering module and is used to repeatedly calculate the multi-frame geographical location of each target and obtain the final positioning result of the target based on the multi-frame geographical location of each target.

[0014] Compared with the prior art, the advantages of the present invention are as follows: real-time target positioning is achieved by fusing target images and lidar point cloud data; the target positioning accuracy is further improved by clustering based on multi-frame target positioning results, and the final true geographical location of the target is obtained, effectively solving the problems of inaccurate target positioning and poor real-time performance during patrol surveys. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a target recognition and localization method based on a panoramic camera and lidar according to the present invention. Figure 2 A schematic diagram of the rectangular recognition area of ​​the present invention; Figure 3 This is a schematic diagram of the structure of a target recognition and positioning system based on a panoramic camera and a lidar according to the present invention. Detailed Implementation

[0016] Referring now to specific embodiments of the invention, examples of which are illustrated in the accompanying drawings. Although the invention will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of the invention as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0017] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Note: The examples described below are merely specific examples and are not intended to limit the embodiments of the present invention to the specific steps, values, conditions, data, order, etc. Those skilled in the art can utilize the concept of the present invention to construct more embodiments not mentioned herein by reading this specification.

[0019] Please see Figure 1 This invention provides a target recognition and localization method based on a panoramic camera and LiDAR. The method includes: Step S100: Calibrate the panoramic camera and LiDAR, acquire the video stream collected by the panoramic camera and LiDAR of the scene to be identified, and generate batch processing frames based on the video stream. The calibration of the panoramic camera and LiDAR includes: Acquire calibration board photos and corresponding radar point cloud data collected by each camera in the panoramic camera and the LiDAR against the calibration board; Based on the calibration board photos and the radar point cloud data, the intrinsic parameters, extrinsic parameters, and distortion parameters of each camera are obtained.

[0020] Specifically, the calibration process is as follows: Step S110: Prepare a black and white calibration board, select one of the cameras in the panoramic camera (in this embodiment, 6 cameras are set), and collect calibration board photos and corresponding radar point cloud data at different distances and angles.

[0021] Step S120: Divide the calibration board into four quadrants, acquire clear images captured at each quadrant angle, collect point cloud data corresponding to the selected photos, evaluate the quality of the currently acquired images, and retake the photos if the calibration requirements are not met until the requirements are met.

[0022] Step S130: Transform the 2D coordinate system (u, v) of the pixels acquired by the panoramic camera to the 3D point cloud coordinate system (x, y, z) of the LiDAR. This allows the acquisition of the camera's intrinsic and extrinsic parameters and distortion parameters. The calculation formula is as follows: ; Where (u, v) are pixel coordinates; , The equivalent focal length in the x and y directions; , (x, y, z) represents the coordinates of the principal point in the image; (x, y, z) represents the coordinates of the 3D point in the LiDAR coordinate system; R is the rotation matrix, describing the attitude rotation from the LiDAR coordinate system to the camera coordinate system; t is the translation vector, describing the position offset from the LiDAR origin to the camera origin; M is the projection matrix, which integrates intrinsic and extrinsic parameters to complete the mapping from 3D LiDAR points to 2D pixels.

[0023] Step S140: Repeat steps S110-S30 to sequentially obtain the intrinsic parameters, extrinsic parameters, and distortion parameters of the six cameras in the panoramic camera.

[0024] Once calibration is complete, acquire the video streams captured by the panoramic camera and LiDAR of the scene to be identified.

[0025] The step of generating batch processing frames based on the video stream includes: Step S150: Based on the nvurisrcbin component in the DeepStream software, the video stream is separated into video data and audio data; Step S160: Generate batch processing frames from the video data using the nvstreammux component in the DeepStream software.

[0026] Specifically, nvurisrcbin is a "URI video source wrapper" component (GStreamer bin) provided by NVIDIA DeepStream. It encapsulates the work of opening network / file streams, decapsulating / de-payloading, hardware decoding, timestamps and reconnection, and outputs GPU zero-copy video frames that can be directly fed to nvstreammux downstream.

[0027] nvstreammux uses a batching mechanism to combine frames from multiple video streams into a single batch of data according to certain rules. Therefore, by using the nvstreammux component to process multiple video streams, dynamically adjusting the batch size, multi-GPU processing, and resolution, it can be combined into a batch of frames.

[0028] Step S200: Based on the target recognition model, target recognition is performed on the batch processing frames to obtain target point cloud data of the same target in the lidar video stream for each camera video stream in the panoramic camera, including: The YOLO-based target recognition model is converted from pt to onnx to generate the engine model; Based on the engine model, target recognition is performed on the batch processing frames, and the identified targets are marked with IDs to obtain target recognition frames. Based on the target recognition frames, target point cloud data of the same target in the lidar video stream of each camera video stream in the panoramic camera is obtained.

[0029] Specifically, in this embodiment of the invention, the YOLO-based target recognition model refers to a model that uses the YOLO (You Only Look Once) series of algorithms to perform "target detection": it performs forward reasoning from the entire image once and directly outputs the bounding box, category, and confidence score of each target.

[0030] The engine model usually refers to the TensorRT serialization engine file, which is a binary file that can be directly loaded and executed on the TensorRT Runtime after graph optimization, operator selection, and precision setting for the target and input shapes.

[0031] Therefore, based on the engine model, target recognition is performed on the batch processing frames. The nvtracker component is used to continuously track the identified targets. The identified and tracked targets are uniquely labeled with IDs to obtain target recognition frames. Finally, based on the target recognition frames, the target point cloud data corresponding to the same target in the lidar video stream of each camera video stream in the panoramic camera is obtained.

[0032] Step S300: Based on the visible area parameters of each camera in the panoramic camera, crop the target point cloud data corresponding to each camera, including: Obtain the visible area parameters of each camera in the panoramic camera, including the horizontal and vertical parameters of the visible area; Based on the horizontal and vertical parameters of the visible area of ​​each camera, the point cloud region corresponding to each camera is obtained. The target point cloud data corresponding to each camera is cropped based on the point cloud region.

[0033] Step S400: For the target point cloud data corresponding to each cropped camera, calculate the pixel coordinates of the target point cloud data on the two-dimensional image based on the calibration results, including: Step S410: For the target point cloud data corresponding to each cropped camera, the three-dimensional points of the target point cloud data are transformed from the world coordinate system to the camera coordinate system based on the extrinsic parameters of the calibration results. The method for transforming a 3D point (x, y, z) from the world coordinate system to the camera coordinate system using extrinsic parameters is shown in the following equation: ; In the formula, These are the non-homogeneous coordinates of a three-dimensional point in the world coordinate system.

[0034] Step S420: Transfer the 3D points in the camera coordinate system Projecting onto the normalized image plane (z=1) yields distortion-free two-dimensional coordinates. as follows: ; Step S430: Correct the two-dimensional coordinates based on the distortion parameters of the calibration results to obtain corrected coordinates. as follows: ; In the formula, Radial distortion coefficient; The tangential distortion coefficient; It is the square of the distance from the normalized coordinates to the origin.

[0035] Step S440: Based on the intrinsic parameter K of the calibration result, convert the corrected coordinates into pixel coordinates on the two-dimensional image. as follows: ; In the formula, the intrinsic parameter matrix is , The focal length (in pixels) is the focal length in the x and y directions. Principal point coordinates (pixel coordinates of the image center).

[0036] Step S500: Cluster all pixel coordinates of the target corresponding to each camera channel to obtain the geographical location of the target in the current frame, including: Step S510: For each camera, record the coordinates of all pixels of the target. 3D point coordinates of the target point cloud data The data is then merged to obtain the merged data. ; Step S520: Generate based on all pixel coordinates of the target Rectangular recognition area in format In the fused data Extract the rectangular recognition region Point cloud collection within; see also Figure 2 As shown, the structure The rectangular area corresponding to the upper left and lower right corners.

[0037] Step S530: Based on a density-based spatial clustering algorithm, the coordinates of the target center point with the highest density in the point cloud set are calculated using a weighted average method. ; Specifically, the center point with the highest density refers to the "representative center" of the cluster with the highest density among all high-density clusters (e.g., the largest number of points per unit area or the largest sum of local density). This center is calculated using a weighted average, so that points with higher density have a greater influence on the center's location.

[0038] First, the "local density" of each data point needs to be quantified. High-density regions are identified by dividing the data into clusters based on density reachability or density peak rules. Then, for each high-density cluster, a weighted average is used to calculate its center coordinates, with the weight being the local density of that point. Finally, the densities of all clusters are compared, and the cluster with the highest density is selected. The weighted center point of this cluster is the coordinate of the target center point with the highest density.

[0039] Step S540: Calculate the distance d and angle γ of the target center point relative to the RTK device; ; Step S550: Based on the distance, the angle, and the latitude and longitude coordinates of the RTK device... Calculate the geographical location of each target along the heading angle α. as follows: ; Wherein, the direction angle is: ; The central angle corresponding to the distance between the target center point coordinates and the RTK device: ; The difference in longitude is: ; In the formula, and These represent the distance and angle of the target center point relative to the RTK device, respectively; R is the Earth's radius; ( ) represents the latitude and longitude coordinates of the RTK device; The heading angle of the RTK device; The target's geographical location; In order to put Convert to radians; To convert the direction angle α to radians.

[0040] Step S600: Repeatedly calculate the multi-frame geographic location of each target, and obtain the final positioning result of the target based on the multi-frame geographic location of each target, including: Step S610: The distance-weighted clustering algorithm clusters the multi-frame geographic locations of each target separately to obtain the localization result of each target. Specifically, the coordinates of a target in one frame from each of the previous steps S100-S500 are obtained by repeating the above steps to obtain the geographical location of the target in multiple frames from each of the six cameras.

[0041] Multi-frame geolocation refers to the location output of the same target at multiple consecutive time points (frames). By amplifying the similarity of nearest neighbor points (the same target) through distance weighting and suppressing the interference of distant points, it ultimately achieves accurate grouping of multi-frame data and outputs reliable target locations.

[0042] Step S620: The method for averaging the positioning results of all road targets to obtain the final positioning result of the target is shown in the following formula: ; In the formula, The final location result of the target; n is the number of cameras in the panoramic camera; This represents the location result of the target corresponding to the nth camera.

[0043] Due to the limited field of view of the cameras, not all cameras can identify the same target. Here, n is the number of cameras that can identify the same target.

[0044] See also Figure 3 As shown, this embodiment of the invention also provides a target recognition and positioning system based on a panoramic camera and lidar, comprising: The calibration module is used to calibrate the panoramic camera and the LiDAR, acquire the video streams collected by the panoramic camera and the LiDAR of the scene to be identified, and generate batch processing frames based on the video streams. The target recognition module is communicatively connected to the calibration module and is used to perform target recognition on the batch processing frames based on the target recognition model to obtain the target point cloud data of each camera video stream in the panoramic camera for the same target in the lidar video stream. The cropping module is communicatively connected to the target recognition module and is used to crop the target point cloud data corresponding to each camera according to the visible area parameters of each camera in the panoramic camera. The coordinate transformation module is communicatively connected to the cropping module and is used to calculate the pixel coordinates of the target point cloud data on the two-dimensional image based on the calibration results for the target point cloud data corresponding to each cropped camera. A clustering module, communicatively connected to the coordinate transformation module, is used to cluster all pixel coordinates of the target corresponding to each camera to obtain the geographical location of each target; and, The final positioning module is communicatively connected to the clustering module and is used to repeatedly calculate the multi-frame geographical location of each target and obtain the final positioning result of the target based on the multi-frame geographical location of each target.

[0045] In summary, the beneficial effects of this invention are as follows: It highly integrates a six-lens panoramic camera, high-precision GNSS, LiDAR, and an edge computing box. The panoramic camera acquires video information from all directions, and at the edge, a recognition algorithm extracts multi-type target information. The target image and LiDAR point cloud data are fused and calculated to achieve real-time target localization. Based on multi-frame target localization result clustering and a multi-lens camera target localization averaging algorithm, the target localization accuracy is further improved, obtaining the target's final true geographical location and effectively solving the problems of inaccurate target localization and poor real-time performance during patrol surveys.

[0046] Specifically, this embodiment corresponds one-to-one with the above method embodiments. The functions of each module have been described in detail in the corresponding method embodiments, so they will not be repeated here.

[0047] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements all or part of the method steps of the above method.

[0048] The present invention can implement all or part of the processes in the above methods, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0049] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory and a processor. The memory stores a computer program that runs on the processor. When the processor executes the computer program, it implements all or part of the method steps described above.

[0050] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting all parts of the computer device through various interfaces and lines.

[0051] Memory can be used to store computer programs and / or modules. The processor performs various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function (e.g., sound playback, image playback, etc.); the data storage area can store data created based on the use of the mobile phone (e.g., audio data, video data, etc.). Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital (SD) cards, Flash Cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0052] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, servers, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0053] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), servers, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A panoramic camera and lidar-based target recognition positioning method, characterized in that, include: The panoramic camera and LiDAR are calibrated, and the video streams captured by the panoramic camera and LiDAR of the scene to be identified are obtained. Batch processing frames are generated based on the video streams. Based on the target recognition model, target recognition is performed on the batch processing frames to obtain the target point cloud data of the same target in the lidar video stream for each camera video stream in the panoramic camera; Based on the visible area parameters of each camera in the panoramic camera, the target point cloud data corresponding to each camera is cropped. For the target point cloud data corresponding to each camera after cropping, the pixel coordinates of the target point cloud data on the two-dimensional image are calculated based on the calibration results. Cluster all pixel coordinates of the target corresponding to each camera to obtain the geographical location of the target in the current frame; The multi-frame geographic location of each target is obtained by repeated calculation, and the final positioning result of the target is obtained based on the multi-frame geographic location of each target. 2.The panoramic camera and lidar based target recognition positioning method of claim 1, wherein, The calibration of the panoramic camera and LiDAR includes: Acquire calibration board photos and corresponding radar point cloud data collected by each camera in the panoramic camera and the LiDAR against the calibration board; Based on the calibration board photos and the radar point cloud data, the intrinsic parameters, extrinsic parameters, and distortion parameters of each camera are obtained. 3.The panoramic camera and lidar based target recognition positioning method of claim 1, wherein, The step of generating batch processing frames based on the video stream includes: The video stream is separated into video data and audio data using the nvurisrcbin component in the DeepStream software. The video data is used to generate batch frames based on the nvstreammux component in the DeepStream software.

4. The target recognition and localization method based on panoramic camera and lidar as described in claim 1, characterized in that, The step of performing target recognition on the batch of frames based on the target recognition model to obtain target point cloud data of the same target in the lidar video stream for each camera video stream in the panoramic camera, including: The YOLO-based target recognition model is converted from pt to onnx to generate the engine model; Based on the engine model, target recognition is performed on the batch processing frames, and the identified targets are marked with IDs to obtain target recognition frames. Based on the target recognition frames, target point cloud data of the same target in the lidar video stream of each camera video stream in the panoramic camera is obtained.

5. The target recognition and localization method based on panoramic camera and lidar as described in claim 1, characterized in that, The step of cropping the target point cloud data corresponding to each camera based on the visible area parameters of each camera in the panoramic camera includes: Obtain the visible area parameters of each camera in the panoramic camera; Based on the visible area parameters of each camera, the point cloud region corresponding to each camera is obtained. The target point cloud data corresponding to each camera is cropped based on the point cloud region.

6. The target recognition and localization method based on panoramic camera and lidar as described in claim 1, characterized in that, The step of calculating the pixel coordinates of the target point cloud data on the two-dimensional image based on the calibration results for the target point cloud data corresponding to each cropped camera includes: For the target point cloud data corresponding to each camera after cropping, the three-dimensional points of the target point cloud data are transformed from the world coordinate system to the camera coordinate system based on the extrinsic parameters of the calibration results. Projecting three-dimensional points in the camera coordinate system onto the normalized image plane yields two-dimensional coordinates. The two-dimensional coordinates are corrected based on the distortion parameters of the calibration results to obtain the corrected coordinates; The calibration coordinates are converted into pixel coordinates on a two-dimensional image based on the intrinsic parameters of the calibration results.

7. The target recognition and localization method based on panoramic camera and lidar as described in claim 1, characterized in that, The process of clustering all pixel coordinates of the target corresponding to each camera to obtain the geographical location of each target includes: For each camera, the pixel coordinates of the target are fused with the 3D point coordinates of the target point cloud data to obtain fused data; A rectangular recognition region is generated based on all pixel coordinates of the target, and a point cloud set within the rectangular recognition region is extracted from the fused data. A density-based spatial clustering algorithm is used to calculate the coordinates of the target center point with the highest density in the point cloud set using a weighted average method. Calculate the distance and angle of the target center point relative to the RTK device; The geographical location of each target is calculated based on the distance, the angle, and the latitude, longitude coordinates and heading angle of the RTK device.

8. The target recognition and localization method based on panoramic camera and lidar as described in claim 7, characterized in that, The method for calculating the geographical location of each target based on the distance, the angle, and the latitude, longitude coordinates and heading angle of the RTK device is shown in the following formula: ； Wherein, the direction angle is: ; The central angle corresponding to the distance between the target center point coordinates and the RTK device: ; The difference in longitude is: ； In the formula, and These represent the distance and angle of the target center point relative to the RTK device, respectively; R is the Earth's radius; ( ) represents the latitude and longitude coordinates of the RTK device; The heading angle of the RTK device; The target's geographical location; In order to put Convert to radians; To convert the direction angle α to radians.

9. The target recognition and localization method based on panoramic camera and lidar as described in claim 1, characterized in that, The process of obtaining the final location result of the target based on the multi-frame geographic location of each target includes: The distance-weighted clustering algorithm clusters the geographic locations of each target across multiple frames to obtain the localization result for each target. The method for averaging the location results of all road targets to obtain the final location result of the target is shown in the following formula: ； In the formula, The final location result of the target; n is the number of cameras in the panoramic camera; This represents the location result of the target corresponding to the nth camera.

10. A target recognition and positioning system based on a panoramic camera and lidar, characterized in that, include: The calibration module is used to calibrate the panoramic camera and the LiDAR, acquire the video streams collected by the panoramic camera and the LiDAR of the scene to be identified, and generate batch processing frames based on the video streams. The target recognition module is communicatively connected to the calibration module and is used to perform target recognition on the batch processing frames based on the target recognition model to obtain the target point cloud data of each camera video stream in the panoramic camera for the same target in the lidar video stream. The cropping module is communicatively connected to the target recognition module and is used to crop the target point cloud data corresponding to each camera according to the visible area parameters of each camera in the panoramic camera. The coordinate transformation module is communicatively connected to the cropping module and is used to calculate the pixel coordinates of the target point cloud data on the two-dimensional image based on the calibration results for the target point cloud data corresponding to each cropped camera. A clustering module, communicatively connected to the coordinate transformation module, is used to cluster all pixel coordinates of the target corresponding to each camera to obtain the geographical location of each target; and, The final positioning module is communicatively connected to the clustering module and is used to repeatedly calculate the multi-frame geographical location of each target and obtain the final positioning result of the target based on the multi-frame geographical location of each target.

Images