Lidar and camera fusion detection method and device, electronic equipment and medium

Abstract

Embodiments of the present application provide a kind of laser radar and camera fusion detection method, device, electronic equipment and medium, involve multimodal target perception fusion technical field.The method comprises: obtaining timestamp alignment camera image data and radar point cloud data, based on camera image data, obtain two-dimensional image detection result, including image detection target, based on radar point cloud data, determine the depth information corresponding to image detection target, obtain three-dimensional image detection result, based on radar point cloud data, obtain radar detection result, including radar detection target, three-dimensional image detection result and radar detection result are fused, and target detection result is obtained.The depth information of image detection target is determined by radar data in the embodiments of the present application, three-dimensional image detection result is obtained, and three-dimensional image detection result and radar detection result are fused, target detection can also be realized in the scene of lower computing power, and more accurate target detection result is obtained.

Description

Technical Field

[0001] This application relates to the field of multimodal target perception fusion technology. Specifically, this application relates to a detection method, device, electronic device, and medium that fuses lidar and camera. Background Technology

[0002] With the rapid development of artificial intelligence, big data and Internet of Things technologies, the control systems of industrial robots are becoming increasingly intelligent, mainly through sensors to perceive the surrounding environment. However, each sensor has its limitations, and a single sensor can only reflect part of the environmental information. In order to improve the effectiveness and stability of the entire system, multi-sensor information fusion has become an inevitable requirement.

[0003] Existing target detection methods process visual and laser information simultaneously during the feature extraction stage to enhance the richness of perceived information. However, while pursuing high accuracy and real-time performance, this strategy also imposes extremely stringent requirements on the computing power of the control system. When faced with situations where computing resources are limited, such as embedded platforms, the above methods cannot be implemented and cannot meet the target detection requirements. Summary of the Invention

[0004] This application provides a detection method, device, electronic device, and medium that integrates LiDAR and camera, aiming to solve the technical problem of inaccurate target detection in scenarios with low computing power, such as embedded platforms.

[0005] Firstly, a detection method integrating lidar and camera is provided, the method comprising:

[0006] Acquire timestamp-aligned camera image data and radar point cloud data;

[0007] Based on camera image data, two-dimensional image detection results are obtained; the two-dimensional image detection results include the image detection target;

[0008] Based on radar point cloud data, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0009] Radar detection results are obtained based on radar point cloud data; the radar detection results include radar-detected targets.

[0010] The target detection result is obtained by fusing the 3D image detection result and the radar detection result.

[0011] Optionally, based on radar point cloud data, the depth information corresponding to the image detection target is determined to obtain the three-dimensional image detection result, including:

[0012] Determine the center point of the target image;

[0013] Based on radar point cloud data and image center points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0014] Optionally, based on radar point cloud data and image center points, the depth information corresponding to the image detection target is determined to obtain the three-dimensional image detection result, including:

[0015] Based on the preset radar resolution, the radar point cloud data is divided into multiple voxel grids;

[0016] Determine the voxel coordinates of each voxel mesh;

[0017] Based on the image center point and voxel coordinates, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0018] Optionally, based on the image center point and voxel coordinates, the depth information corresponding to the image detection target is determined to obtain the three-dimensional image detection result, including:

[0019] Projecting the image center point from the two-dimensional pixel coordinate system to the unit sphere coordinate system yields the image projection point of the image center point in the unit sphere coordinate system. The unit sphere coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin, and includes a unit sphere centered on the camera optical center.

[0020] Project each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system.

[0021] Based on image projection points and radar projection points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0022] Optionally, before projecting each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system, the following steps are included:

[0023] Project each voxel coordinate point from the three-dimensional radar coordinate system to the two-dimensional pixel coordinate system to obtain the radar pixel point of the voxel coordinate point in the two-dimensional pixel coordinate system.

[0024] Determine the pixel distance between each radar pixel and the image center point;

[0025] Radar pixels whose pixel distance is less than a preset distance threshold are designated as target radar pixels.

[0026] Determine the target voxel grid corresponding to the target radar pixel;

[0027] Projecting each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system yields the radar projection points of the voxel coordinate points in the unit spherical coordinate system, including:

[0028] Projecting the target voxel coordinates from the three-dimensional radar coordinate system to the unit spherical coordinate system yields the radar projection points of the target voxel coordinates in the unit spherical coordinate system.

[0029] Optionally, based on image projection points and radar projection points, the depth information corresponding to the image detection target is determined to obtain the three-dimensional image detection result, including:

[0030] Determine the spatial distance between each radar projection point and the image projection point in the unit spherical coordinate system;

[0031] Based on spatial distance, a preset number of candidate radar points are selected from multiple radar projection points;

[0032] Based on preset screening criteria, at least three target radar points are selected from the candidate radar points;

[0033] Based on at least three target radar points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0034] Optionally, based on preset filtering criteria, at least three target radar points are selected from the candidate radar points, including:

[0035] Based on spatial distance, candidate radar points are arranged from smallest to largest;

[0036] Multiple candidate groups are obtained by dividing the radar points in the order of arrangement; each candidate group includes at least three candidate radar points.

[0037] Determine the distance between any two candidate radar points in the candidate group in turn;

[0038] Candidate groups whose distances meet the preset screening criteria are selected as target radar groups;

[0039] The candidate radar points in the target radar group are used as the target radar points.

[0040] Optionally, based on at least three target radar points, the depth information corresponding to the image detection target is determined to obtain the three-dimensional image detection result, including:

[0041] Determine the target plane based on at least three target radar points;

[0042] Based on the distance between the target plane and the origin of the unit spherical coordinate system, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0043] Optionally, acquire timestamp-aligned camera image data and radar point cloud data, including:

[0044] Obtain camera image data and corresponding image timestamps;

[0045] Acquire radar point cloud data and the corresponding radar timestamps;

[0046] Align the image timestamps and radar timestamps to obtain timestamp-aligned camera image data and radar point cloud data.

[0047] Optionally, the radar detection results include the radar detection bounding box corresponding to the radar detected target;

[0048] The target detection results are obtained by fusing the 3D image detection results and the radar detection results, including:

[0049] The image center point of the target is mapped to the three-dimensional radar coordinate system to obtain the reference projection point of the image center point in the three-dimensional radar coordinate system.

[0050] If the reference projection point is within the radar detection box, then determine the first probability of the radar detection result and the second probability of the 3D image detection result; compare the first probability and the second probability, and take the detection result with the higher probability as the target detection result;

[0051] If the reference projection point is outside the radar detection frame, the 3D image detection result will be used as the target detection result.

[0052] Secondly, a detection device integrating lidar and camera is provided, the device comprising:

[0053] The data acquisition module is used to acquire timestamp-aligned camera image data and radar point cloud data;

[0054] The image detection module is used to obtain two-dimensional image detection results based on camera image data; the two-dimensional image detection results include the image detection target;

[0055] The depth acquisition module is used to determine the depth information corresponding to the image detection target based on radar point cloud data, and obtain the three-dimensional image detection result;

[0056] The radar detection module is used to obtain radar detection results based on radar point cloud data; the radar detection results include radar-detected targets.

[0057] The result fusion module is used to fuse the 3D image detection results and the radar detection results to obtain the target detection results.

[0058] Thirdly, an electronic device is provided, the electronic device comprising:

[0059] A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods in the first aspect of this application.

[0060] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the detection method of fusion of lidar and camera as shown in any of the first aspects of this application.

[0061] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any one of the methods in the first aspect of this application.

[0062] The beneficial effects of the technical solutions provided in this application are:

[0063] The detection method based on the fusion of LiDAR and camera provided in this application obtains the image detection target by obtaining the two-dimensional image detection result from the camera image data, calculates the depth information of the image detection target in three-dimensional space based on the LiDAR point cloud data, thereby obtaining the three-dimensional image detection result of the image detection target, and obtains the radar detection target based on the radar detection result of the radar point cloud data. The three-dimensional image detection result and the radar detection result are fused to obtain the target detection result. The method uses a lightweight detection algorithm to obtain the detection results in different dimensions, and upgrades the two-dimensional image detection result based on the radar data, thereby obtaining the target detection result by fusing the results. It is suitable for scenarios with low computing power, such as those that do not support complex radar detection, and can obtain accurate target detection results even with limited computing power. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0065] Figure 1 A schematic diagram illustrating an application scenario of a detection method that combines lidar and camera, as provided in an embodiment of this application.

[0066] Figure 2 A schematic flowchart illustrating a detection method that fuses lidar and camera data, provided in an embodiment of this application;

[0067] Figure 3 This is a schematic diagram illustrating the process of obtaining depth information in a detection method that combines lidar and camera, as provided in an embodiment of this application.

[0068] Figure 4 A schematic diagram of a voxel grid in a detection method that fuses lidar and camera according to an embodiment of this application;

[0069] Figure 5A schematic diagram of the projection of unit sphere coordinates in a detection method that fuses lidar and camera according to an embodiment of this application;

[0070] Figure 6 A schematic diagram of the radar coordinate system in a detection method that fuses lidar and camera according to an embodiment of this application;

[0071] Figure 7 A schematic diagram of the target voxel grid in a detection method that combines lidar and camera according to an embodiment of this application;

[0072] Figure 8 This is a schematic diagram illustrating the calculation of depth information in a detection method that fuses lidar and camera, as provided in an embodiment of this application.

[0073] Figure 9 A schematic flowchart illustrating an example of a detection method that fuses lidar and camera, provided in an embodiment of this application;

[0074] Figure 10 A schematic diagram of a detection device that fuses lidar and camera, provided in an embodiment of this application;

[0075] Figure 11 This is a schematic diagram of the structure of an electronic device to which a detection method combining lidar and camera is applicable, as provided in an embodiment of this application. Detailed Implementation

[0076] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0077] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by this art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The terms “or,” “and / or,” “including at least one of the following,” etc., as used in this application, can be interpreted as inclusive, or mean any one or any combination thereof.

[0078] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0079] In the specific embodiments of this application, any data related to the object, such as data involved in the use of the application, is required. When the embodiments of this application are applied to specific products or technologies, permission or consent from the object is required, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if any of the aforementioned object-related data is involved in the embodiments of this application, this data must be obtained with the object's authorization and consent, and in accordance with the relevant laws, regulations, and standards of the country and region.

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0081] First, the technical terms used in this application will be introduced and explained:

[0082] Radar point cloud data: Radar point cloud data consists of a series of radar points in three-dimensional space collected by a radar system. Each radar point typically has at least three coordinates, which define the point's position in three-dimensional space. Radar point clouds are commonly used in remote sensing, surveying, autonomous vehicles, and robot navigation to create three-dimensional models of the environment.

[0083] Depth information refers to the distance of an object or point from an observer or reference plane in three-dimensional space. It helps determine the position and shape of an object in three-dimensional space. This information is crucial in fields such as computer vision, robotics, virtual reality (VR), augmented reality (AR), and autonomous driving. In computer vision and robotics, the acquisition and analysis of depth information are essential for tasks such as navigation, obstacle avoidance, interaction, and 3D reconstruction.

[0084] Voxel: A voxel is short for volume pixel, the smallest unit in three-dimensional space, similar to a pixel in two-dimensional space. Voxels are used to represent three-dimensional data and are commonly used in scientific and engineering fields.

[0085] In existing technologies, single-sensor-based target detection methods often suffer from insufficient detection accuracy due to environmental changes and sensor field of view, leading to missed or false detections. To significantly improve the accuracy of target recognition for vehicles, pedestrians, and obstacles in operational environments, the industry generally favors multi-sensor fusion algorithms. These algorithms process visual and laser information simultaneously during feature extraction, enriching the perceived information to adapt to diverse application needs. However, this approach requires substantial computing resources. In practical applications such as embedded platforms for mobile robots in industrial environments and large-scale mass production solutions, the deployment environment is often limited by embedded platforms with lower computing power, preventing these algorithms from running in real-time and failing to meet the performance requirements for target detection in industrial settings.

[0086] The detection method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this application, which integrates lidar and camera, are intended to solve at least one of the above-mentioned technical problems in the prior art.

[0087] To address at least one of the aforementioned technical problems or areas requiring improvement in related technologies, this application proposes a detection method, apparatus, electronic device, and medium that fuses LiDAR and camera data. The LiDAR and camera fusion detection method provides the following steps: First, it obtains a two-dimensional image detection result of the target from camera image data. Then, it calculates the depth information of the target in three-dimensional space based on LiDAR point cloud data, thus obtaining a three-dimensional image detection result of the target. Finally, it obtains a radar detection target based on the radar point cloud data. The three-dimensional image detection result and the radar detection result are then fused to obtain the target detection result. A lightweight detection algorithm is used to acquire detection results in different dimensions, and the two-dimensional image detection result is upgraded based on radar data. The fusion result yields the target detection result. This method is suitable for scenarios with low computing power or those that do not support complex radar detection, and can obtain accurate target detection results even with limited computing power.

[0088] Furthermore, a three-dimensional unit spherical coordinate system is introduced. When determining the depth information of the image detection target, the radar points corresponding to the radar point cloud data are projected onto the unit spherical coordinate system, and the image center point corresponding to the camera image data is also projected onto the unit spherical coordinate system. In the unit spherical coordinate system, the depth information corresponding to the image detection target is obtained, which can determine the spatial distance between each point and obtain more accurate depth information, making the subsequent detection results more accurate.

[0089] Furthermore, based on the above steps, the 3D image detection results and radar detection results can be obtained, and their consistency can be verified and aligned. Under certain conditions, the two detection results can be fused together, avoiding false detections from a single detection result and missed detections in blind spots of the camera and radar fields of view, thus increasing the comprehensiveness and accuracy of the detection results.

[0090] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0091] Figure 1 This is a schematic diagram of an application scenario for the detection method that combines lidar and camera according to an embodiment of this application. The application environment may include a machine device 100, on which a radar device 200 and a camera device 300 are installed.

[0092] Specifically, the machine device 100 acquires timestamp-aligned camera image data and radar point cloud data through the radar device 200 and the camera device 300. Based on the camera image data, a two-dimensional image detection result is obtained. Based on the radar point cloud data, the depth information corresponding to the image detection target is determined to obtain a three-dimensional image detection result. Based on the radar point cloud data, a radar detection result is obtained. The three-dimensional image detection result and the radar detection result are fused to obtain the target detection result. The two-dimensional image detection result includes the image detection target, and the radar detection result includes the radar detection target.

[0093] In practical implementation, when the above method is applied in the scenario of industrial robot development, the machine equipment 100 can be an industrial robot. The computing power of this equipment is generally small, making it difficult to support functions with high computing power requirements such as radar detection. Using the detection method of LiDAR and camera fusion of this application improves the target detection accuracy and real-time performance in this scenario. It does not require the computing power for complex analysis of radar data. Even with low computing power and limited computing resources, it can determine the depth information of the camera image detection results and complete the target detection, thereby reducing the hardware cost of industrial robots and promoting the mass production of intelligent detection robots.

[0094] In specific implementation, the aforementioned machinery and equipment 100 may include industrial robot equipment, service robot equipment, sweeping robot equipment, and agricultural robot equipment, etc.; radar equipment 200 may include lidar transmitters and radar sensors, etc.; and camera equipment 300 may include cameras, ordinary cameras, high-speed cameras, and infrared cameras, etc., and may be configured according to specific needs.

[0095] The above application scenario is just an example and does not limit the application scenarios of the detection method of LiDAR and camera fusion in this application.

[0096] In some possible implementations, taking a machine as the executing entity as an example, embodiments of this application provide a detection method that fuses lidar and camera data, such as... Figure 2 As shown, it may include the following steps:

[0097] S210 acquires timestamp-aligned camera image data and radar point cloud data.

[0098] Among them, radar point cloud data can be obtained in advance through lidar scanning, and camera image data can be obtained through camera acquisition.

[0099] The detection target can be objects or people in the surrounding environment, and the specific target can be determined based on the actual scene in which the aforementioned machine is located. There are no restrictions here.

[0100] Specifically, the camera captures the surrounding environment to obtain camera image data, and obtains radar point cloud data aligned with the timestamp of the camera image data. There is at least one detection target in the surrounding environment. For each detection target, the detection result can be obtained by using the above-mentioned detection method of fusion of lidar and camera, so as to quickly detect objects in the surrounding environment for corresponding processing or operation.

[0101] In the specific implementation process, single-source target detection is performed through a camera to obtain the two-dimensional target detection result of the image, thereby obtaining the detected target in the image and acquiring the radar point cloud data corresponding to the image. Based on the radar point cloud data, the depth information corresponding to the detected target in the image is determined, thus obtaining the three-dimensional image detection result. The three-dimensional image detection result is combined with the three-dimensional radar detection result to obtain the target detection result. Since this application does not require feature-level data fusion, a lightweight target detection model can be used to achieve real-time detection on a development platform with limited resources. The target detection step can use any general target detection algorithm, such as the real-time target detection algorithm YOLO and point cloud target detection PointPillars, without affecting the subsequent fusion of detection results.

[0102] In the specific implementation process, the detection results obtained from multiple sensors can also be fused, such as pressure sensors, temperature sensors, sound sensors, humidity sensors and ultrasonic sensors, etc. This application does not limit this, and the fusion of detection results under multiple modes can improve the detection accuracy while ensuring the real-time performance of the detection algorithm.

[0103] S220 obtains two-dimensional image detection results based on camera image data.

[0104] The two-dimensional image detection result includes the image detection target, and one image detection target corresponds to one two-dimensional image detection box.

[0105] Specifically, target detection is performed on camera image data to obtain two-dimensional image detection results. The image detection results include two-dimensional image information corresponding to the detected target, including the position of the image center point and the size of the image, so that the image center point can be projected to obtain the corresponding depth information.

[0106] In the specific implementation process, after acquiring camera image data, two-dimensional image detection results can be obtained based on the camera image data, and the image detection target in the two-dimensional image detection results can be obtained. The depth information of the image detection target can be recovered based on radar point cloud data, thereby obtaining three-dimensional image detection results. That is, the detection target under the camera's field of view is obtained through two-dimensional image detection, and the depth information is determined based on the camera's detection target using radar point cloud data, which can reduce computing power consumption and computational costs.

[0107] S230, based on radar point cloud data, determines the depth information corresponding to the image detection target and obtains the three-dimensional image detection result.

[0108] Specifically, based on the obtained radar point cloud data, the depth information corresponding to the image detection target is determined, and a three-dimensional image detection result is obtained. Based on the two-dimensional image detection result and the corresponding depth information, the position of the corresponding image detection target in three-dimensional space can be determined. The target detection result obtained in this way can more comprehensively describe the position information of the detected target and can also more accurately identify the detected target.

[0109] In the specific implementation process, the process of obtaining the depth information corresponding to the image detection target based on radar point cloud data is as follows: Figure 3 As shown, the image center point of the target is obtained, the pixel coordinates of the center point are determined, the pixel coordinates are projected onto the unit spherical coordinate system, and the radar point cloud data is processed by volume element meshing to obtain the corresponding voxel mesh. Based on the filtering conditions, the radar point cloud mesh that meets the conditions is selected, the corresponding radar points are determined, and projected onto the unit spherical coordinate system. Based on the projection of the image center point and the projection of the radar points, the depth information corresponding to the target is obtained. Specifically, the two-dimensional image detection box corresponding to the target can be determined, and the depth information of the two-dimensional image detection box can be recovered (i.e., the 2D detection box depth information recovery shown in the figure). The unit spherical coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin.

[0110] In practical implementation, the two-dimensional target detection method based on visual images takes an image as input and can obtain the image detection target, the category of the image detection target, and the pixel coordinates (u, v) of the center point of the image detection target in the image. It can determine the two-dimensional image detection box corresponding to the image detection target and obtain information such as the length and width of the image detection box. However, camera image data can only obtain two-dimensional information of the detected target in the environment and cannot obtain the actual distance to the detected target. Therefore, it cannot provide a reliable reference for the path planning and obstacle avoidance functions of mobile robots or other carriers. Moreover, when the camera acquires the image, it will reduce the three-dimensional information of the object to two-dimensional image information. In order to better fuse the two-dimensional detection results of the image and the three-dimensional detection results of the radar, this application proposes a depth recovery method for two-dimensional target detection results. The two-dimensional image information is back-projected onto the unit spherical coordinate system to obtain depth information. Combined with the radar point cloud information, the depth information, length, width, height and azimuth information of the center point of each two-dimensional image are recovered.

[0111] In the specific implementation process, such as Figure 3 As shown, after obtaining depth information, the length, width, height, and azimuth information of the image-detected target can be calculated. Based on the information corresponding to the image-detected target, a three-dimensional target detection box (i.e., the 3D target detection box shown in the figure) is generated. The steps include: converting all the center points of the two-dimensional detection results into three-dimensional center points in the radar coordinate system; determining a range with the three-dimensional center points as the center and a preset length as the radius, such as 3 meters; clustering the radar point cloud data within this range; the largest cluster is the three-dimensional target cluster; then calculating the difference between the maximum and minimum values ​​of the point cloud clusters in the x, y, and z directions to obtain the length, width, and height information of the three-dimensional target detection box; comparing the obtained length, width, and height information with the preset length, width, and height parameters corresponding to the target category; and selecting the smaller value as the final length, width, and height. In addition, based on algorithms such as principal component analysis (PCA), the direction vector of the radar point cloud clusters can be obtained as the direction of the target.

[0112] In the specific implementation process, two-dimensional image detection results are obtained based on camera image data. The two-dimensional image detection results are back-projected and the corresponding depth information is restored to obtain three-dimensional image detection results. The three-dimensional image detection results are then fused with the three-dimensional radar detection results. Detection results of the same dimension are fused to make the fused detection results more accurate.

[0113] S240 obtains radar detection results based on radar point cloud data.

[0114] The radar detection results include radar-detected targets, and the radar detection results are three-dimensional detection results.

[0115] Specifically, radar point cloud data includes many radar points. By performing operations such as feature extraction, clustering, and model matching on multiple radar points, the corresponding radar detection results and radar detection targets in the radar detection results are obtained. This allows the radar detection targets to be combined with the 3D image detection results to obtain target detection results, which can identify targets more comprehensively and accurately.

[0116] In the specific implementation process, radar point cloud data is acquired and preprocessed, including removing outliers and reducing data volume using filtering methods. After preprocessing, features can be extracted from the radar point cloud to determine its shape, size, radar echo intensity, and local features. Then, the radar point cloud can be segmented into different clusters, which can be matched using predefined 3D graphic models, classified using methods such as random forests and support vector machines, or directly processed using neural networks to obtain the corresponding radar detection results.

[0117] The S250 integrates the 3D image detection results with the radar detection results to obtain the target detection results.

[0118] Specifically, by obtaining the 3D image detection results and radar detection results corresponding to the target, comparing the 3D image detection results and radar detection results, and filling in the gaps in false detections and missed detections, a more comprehensive detection result can be obtained.

[0119] In the specific implementation process, one of the 3D image detection results and the radar detection results can be selected as the detection result based on preset selection conditions. Since both the 3D image detection results and the radar detection results include corresponding probability information, the detection result can be selected based on the probability information. Alternatively, features can be extracted from the 3D image detection results and the radar detection results respectively. Specifically, features such as 3D position, intensity, and shape can be extracted from the radar detection results, and features such as edges, corners, color, and texture can be extracted from the 3D image detection results. The correspondence relationship is used to associate the features in the radar point cloud with the features in the camera image to determine the target detection result. Feature association can use feature matching algorithms such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features).

[0120] In some possible implementations, step S230 above, based on radar point cloud data, determines the depth information corresponding to the image detection target to obtain a three-dimensional image detection result, including:

[0121] Determine the center point of the target image;

[0122] Based on radar point cloud data and image center points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0123] The image center point is the center point of the two-dimensional image corresponding to the target in the camera image data.

[0124] Specifically, the process involves acquiring a two-dimensional image corresponding to the target image, determining the length and width of the two-dimensional image, determining the corresponding image detection box based on the length and width of the image corresponding to the target image, obtaining the center point of the image detection box (i.e., obtaining the image center point of the target), and recovering the depth information corresponding to the image center point based on radar point cloud data to obtain the depth information corresponding to the target image.

[0125] In the specific implementation process, the image detection box corresponding to the image detection target is used as the center point of the image detection box. The position of the image center point in the two-dimensional pixel coordinate system can be determined. The two-dimensional pixel coordinate system takes the upper left corner of the image as the origin, the horizontal rightward as the x-axis, and the vertical downward as the y-axis. The coordinates of the point in the two-dimensional pixel coordinate system are (u, v).

[0126] In some possible implementations, the above steps, based on radar point cloud data and image center points, determine the depth information corresponding to the image detection target to obtain a three-dimensional image detection result, including:

[0127] Based on the preset radar resolution, the radar point cloud data is divided into multiple voxel grids;

[0128] Determine the voxel coordinates of each voxel mesh;

[0129] Based on the image center point and voxel coordinates, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0130] Here, a voxel is a volume pixel in three-dimensional space. Radar point cloud data includes multiple radar points. One voxel grid can correspond to one or more radar points, and the specifics will vary depending on the corresponding partitioning method.

[0131] Specifically, based on the preset radar resolution, the radar point cloud data is divided into multiple voxel grids. The length, width, and height of each voxel grid correspond to the resolution size, and the voxel coordinates of each voxel grid are the positions of the voxels in the radar coordinate system. Based on the image center point and the voxel coordinates, the depth information corresponding to the image detection target is obtained, thereby obtaining the three-dimensional image detection result.

[0132] In the specific implementation process, the radar point cloud data is processed into a grid, and the radar coordinate system is as follows: Figure 4As shown, the radar equipment emits a lidar sensor to scan. The x-direction represents the forward direction, the y-direction points to the left of the horizontal plane, and the z-direction points upwards perpendicular to the plane defined by x and y. In the forward direction of the radar, the lidar point cloud is divided into several voxel grids according to a preset resolution. For example, assuming the resolution of the voxel grid is 0.16 meters, the entire lidar point cloud can be divided into M × N voxel cylinders of size 0.16m × 0.16m × H, where M, N, and H are the differences between the maximum and minimum values ​​of the point cloud in the y-axis, x-axis, and z-axis directions, respectively.

[0133] In practical implementation, when dividing radar point cloud data into grids, the size of the voxel grid can be set manually, so that one voxel grid can correspond to multiple radar points. The voxel coordinate point corresponding to each voxel grid can be calculated based on multiple radar points within the same grid. The voxel coordinate point can also be called the voxel grid center point.

[0134]

[0135] p center_lidar =(x center_lidar y center_lidar , z center_lidar )

[0136] Where, p center_lidar x represents the coordinates of the center point of the voxel grid in the radar coordinate system. center_lidar y center_lidar and z center_lidar represents the coordinates of the center point of the voxel grid on the x, y, and z axes, respectively; n is the number of radar points in the voxel grid; and i is the index of the radar point. i Let x be the coordinates of each radar point in the voxel grid on the x-axis, and y be the coordinates of each radar point on the voxel grid on the x-axis. i Let z be the coordinates of each radar point in the voxel grid on the y-axis, z be the coordinates of each radar point on the voxel grid. i represents the coordinates of each radar point in the voxel grid on the z-axis.

[0137] In some possible implementations, the above steps determine the depth information corresponding to the image detection target based on the image center point and voxel coordinate points to obtain the three-dimensional image detection result, including:

[0138] Projecting the image center point from the two-dimensional pixel coordinate system to the unit sphere coordinate system yields the image projection point of the image center point in the unit sphere coordinate system.

[0139] Project each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system.

[0140] Based on image projection points and radar projection points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0141] The unit sphere coordinate system is a three-dimensional coordinate system established with the camera's optical center as the origin. The unit sphere coordinate system includes a unit sphere centered on the camera's optical center. The unit sphere coordinate system can also be called the camera coordinate system.

[0142] Specifically, the image center point is projected from the two-dimensional pixel coordinate system to the unit sphere coordinate system to obtain the image projection point of the image center point in the unit sphere coordinate system. Then, each voxel coordinate point is projected from the three-dimensional radar coordinate system to the unit sphere coordinate system to obtain the radar projection point of the voxel coordinate point in the unit sphere coordinate system. Based on the relationship between the image projection point and the radar projection point in the unit sphere coordinate system, the depth information corresponding to the detected target is obtained. The image projection point is located on the surface of the unit sphere in the unit sphere coordinate system.

[0143] In the specific implementation process, the image center point of each two-dimensional target detection box is determined. Each two-dimensional target detection box corresponds to a detection target, and the center point of the two-dimensional target detection box is the image center point of the corresponding detection target. The step of projecting the coordinates of the image center point in the two-dimensional pixel coordinate system to the unit spherical coordinate system may include: setting the pixel coordinates of the center point of the target detection box to p i = (u, v), the steps to backproject the coordinates of the center point to the unit spherical coordinate system are as follows:

[0144] First, project the coordinates in the two-dimensional pixel coordinate system onto the normalized plane:

[0145]

[0146] Where u is the x-coordinate of the image center point in the two-dimensional pixel coordinate system, v is the y-coordinate of the image center point in the two-dimensional pixel coordinate system, and x... p The x-coordinate of the image center point after normalization, y p The normalized ordinate of the image center point, (x p y p , 1) is the coordinate p of the image center point i The corresponding coordinates in the normalized plane, f x f y c x and c y These are camera parameters, obtained from camera calibration.

[0147] Then, map the normalized coordinates to the unit spherical coordinate system:

[0148]

[0149]

[0150] Where, x p The x-coordinate of the image center point after normalization, y p The ordinate of the image center point after normalization, x s Let y be the x-coordinate of the image center point in the unit spherical coordinate system. s The ordinate (z) of the image center point in the unit spherical coordinate system. s The perpendicular coordinate of the image center point in the unit spherical coordinate system, (x s y s , z s () represents the coordinate mapping of the image center point in the unit spherical coordinate system, such as Figure 5 As shown, the center of the sphere O is the optical center of the camera, the radius r = 1, and point R is the center point of the mapped image.

[0151] In the specific implementation process, the voxel coordinates of the voxel mesh are projected onto the unit spherical coordinate system. Then, the 3D radar points in the unit spherical coordinate system are normalized to obtain the corresponding 3D radar projection points. This can be achieved using a rotation matrix R. cl Translation vector t cl Perform the conversion, R cl With t cl The installation position between the camera and radar is related to the parameter calibration.

[0152] p c =R cl ·p l +t cl

[0153] Where, p c p represents the radar projection point projected onto the unit spherical coordinate system. l These are the voxel coordinate points corresponding to the voxel grid in the radar coordinate system.

[0154] In some possible implementations, before projecting each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system, the above steps include:

[0155] Project each voxel coordinate point from the three-dimensional radar coordinate system to the two-dimensional pixel coordinate system to obtain the radar pixel point of the voxel coordinate point in the two-dimensional pixel coordinate system.

[0156] Determine the pixel distance between each radar pixel and the image center point;

[0157] Radar pixels whose pixel distance is less than a preset distance threshold are designated as target radar pixels.

[0158] Determine the target voxel grid corresponding to the target radar pixel;

[0159] Projecting each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system yields the radar projection points of the voxel coordinate points in the unit spherical coordinate system, including:

[0160] Projecting the target voxel coordinates from the three-dimensional radar coordinate system to the unit spherical coordinate system yields the radar projection points of the target voxel coordinates in the unit spherical coordinate system.

[0161] Specifically, each voxel coordinate point is projected from the 3D radar coordinate system to the 2D pixel coordinate system to obtain the position of the radar pixel point in the 2D pixel coordinate system. The pixel distance between each radar pixel point and the image center point is determined. Radar pixels with a pixel distance less than a preset distance threshold are taken as target radar pixels. The target voxel grid corresponding to the target radar pixel point is determined. Then, the target voxel coordinate points of the target voxel grid are projected from the 3D radar coordinate system to the unit spherical coordinate system to obtain the radar projection points of the target voxel coordinate points in the unit spherical coordinate system, so as to calculate and recover the corresponding depth information.

[0162] In the specific implementation process, the voxel coordinate point p center_lidar The formula used to project the two-dimensional pixel coordinate system onto the camera is as follows:

[0163]

[0164] Among them, (u pixel_lidar v pixel_lidar , 1) represents the coordinates of the radar pixel, u pixel_lidar v is the x-coordinate of the radar pixel. pixel_lidar The vertical coordinate of the radar pixel, such as Figure 6 As shown, x center_lidar y center_lidar and z center_lidar T represents the x-axis, y-axis, and z-axis coordinates of the voxel point in the radar coordinate system. lc f is the spatial position transformation matrix for the radar and camera. x f y c x and c y All camera projection parameters are obtained through calibration.

[0165] In the specific implementation process, the radar point cloud data is divided into an M×N voxel grid. The voxel coordinates of each voxel grid are projected onto the pixel coordinate system to obtain M×N radar pixels. Based on the coordinates of the radar pixels and the distance between them and the coordinates of the image center point, the corresponding target radar pixels and target voxel grids are determined. The distance is calculated using the following formula:

[0166]

[0167] Where (u,v) are the coordinates of the image center point, (u pixel_lidar ,v pixel_lidar ) represents the coordinates of a radar pixel, d box-lidar For the pixel distance mentioned above, when d box-lidar When the value is less than a certain threshold D, the corresponding voxel grid is selected as the target voxel grid. The threshold D is an empirical value and is related to the category of the target. The unit is pixels. When the target of the box is a small object such as a person, bicycle, or sign, D is set to a smaller value. When the target of the box is a large object such as a vehicle, D is set to a larger value. The specific setting can be adjusted according to the actual situation.

[0168] In the specific implementation process, such as Figure 7 As shown, the central dot 701 represents the image center point, the gray grid 702 represents the target voxel grid within the distance threshold range, and the circular dashed line 703 represents the threshold range boundary with the distance threshold as the radius.

[0169] In some possible implementations, the above steps determine the depth information corresponding to the image detection target based on image projection points and radar projection points to obtain a three-dimensional image detection result, including:

[0170] Determine the spatial distance between each radar projection point and the image projection point in the unit spherical coordinate system;

[0171] Based on spatial distance, a preset number of candidate radar points are selected from multiple radar projection points;

[0172] Based on preset screening criteria, at least three target radar points are selected from the candidate radar points;

[0173] Based on at least three target radar points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0174] Specifically, the spatial distance between each radar projection point and the image projection point in the unit spherical coordinate system is determined. Based on the spatial distances corresponding to each radar projection point, a preset number of candidate radar points are selected from multiple radar projection points. Based on preset filtering conditions, at least three target radar points are selected from the candidate radar points. The depth information corresponding to the image detection target is calculated using the selected at least three target radar points, thereby obtaining the three-dimensional image detection result.

[0175] In the specific implementation process, the projection point of the image center point in the unit spherical coordinate system is p. s =(x s y s , zs For radar projection points in the unit spherical coordinate system, random sampling is performed to select those with a distance p. s The most recent preset number of candidate radar points are selected. For example, if the preset number is set to 10, the 10 most recent candidate radar points are selected and the corresponding real radar point information is marked for the candidate radar points. Then, at least three target radar points are selected from the candidate radar points, the real radar point information corresponding to the target radar points is determined, and the depth information corresponding to the image detection target is calculated.

[0176] In some possible implementations, the above steps involve selecting at least three target radar points from candidate radar points based on preset screening criteria, including:

[0177] Based on spatial distance, candidate radar points are arranged from smallest to largest;

[0178] Multiple candidate groups were obtained by dividing the candidates sequentially according to their arrangement order;

[0179] Determine the distance between any two candidate radar points in the candidate group sequentially.

[0180] Candidate groups whose distances meet preset screening criteria are selected as target radar groups.

[0181] The candidate radar points in the target radar group are used as the target radar points.

[0182] The candidate group includes at least three candidate radar points.

[0183] Specifically, the spatial distance between each radar projection point and the image projection point in the unit spherical coordinate system is determined. Based on the spatial distance, the candidate radar points are arranged from smallest to largest and divided sequentially according to the arrangement order to obtain multiple candidate groups. The distance between any two candidate radar points in each candidate group is calculated sequentially. The candidate group whose distance meets the preset screening conditions is taken as the target radar group, and the candidate radar points in the target radar group are taken as the target radar points to obtain at least three target radar points, thereby determining the corresponding depth information.

[0184] In the specific implementation process, 10 candidate radar points are acquired and iteratively verified: the 10 radar points are sorted in ascending order of spatial distance and numbered from 1 to 10. The size of the candidate group can be set to 3. Three points are selected in sequence according to distance as a group, for example: (1, 2, 3), (1, 2, 4), (1, 2, 5)... For each candidate group, the distance between candidate radar points is compared pairwise. If the distance between any two points is greater than a preset threshold, such as 2 meters, the group is skipped and the next group of points is calculated. If the distance between any two points is less than the preset threshold, the verification ends, the corresponding candidate group is taken as the target radar group, and the candidate radar points of the target radar group are taken as the target radar points. If no candidate group meets the conditions after the iteration ends, the detection target is abandoned.

[0185] In some possible implementations, the above steps involve determining the depth information corresponding to the image detection target based on at least three target radar points to obtain a three-dimensional image detection result, including:

[0186] Determine the target plane based on at least three target radar points;

[0187] Based on the distance between the target plane and the origin of the unit spherical coordinate system, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0188] Specifically, based on at least three target radar points, the corresponding target plane is determined in the unit spherical coordinate system, and the distance between the target plane and the origin of the unit spherical coordinate system is determined. Based on this distance and the radius of the unit sphere, the depth information corresponding to the image detection target is calculated, thereby obtaining the three-dimensional image detection result.

[0189] In the specific implementation process, such as Figure 8 As shown, a plane, denoted as plane P, is determined based on at least three target radar points. The radius r of a unit sphere is 1. The origin O of the unit sphere coordinate system and the image projection point p are connected. s The line connecting the two points intersects the plane P at point P, where point P represents the true location of the image center point in three-dimensional space, points ABC represent the three target radar points, point p represents the image projection point on the normalized unit sphere, point M represents the projection of point P onto the z-axis, and point m represents the projection of point p onto the z-axis. The depth information OM can be calculated using the following formula:

[0190]

[0191] Where OP is the distance from the origin to point P on the plane, the length of OP can be calculated using the formula for the distance from a point to a plane, the depth information OM corresponding to the center point of the image is the projection distance of OP on the z-axis of the coordinate system, Op is the distance from the origin to point P, Om is the projection distance of point P on the z-axis, and Om = 1.

[0192] In some possible implementations, the steps described above for acquiring timestamp-aligned camera image data and radar point cloud data include:

[0193] Obtain camera image data and corresponding image timestamps;

[0194] Acquire radar point cloud data and the corresponding radar timestamps;

[0195] Align the image timestamps and radar timestamps to obtain timestamp-aligned camera image data and radar point cloud data.

[0196] The process of obtaining a timestamp can also be referred to as collecting a timestamp.

[0197] Specifically, radar point cloud data and corresponding image acquisition timestamps are acquired, as well as camera image data and corresponding radar timestamps. Based on the data acquisition timestamps, the radar point cloud data and the camera image data are aligned with their starting timestamps, thereby ensuring that the target information in the image and radar point cloud is consistent, which helps to improve the accuracy and reliability of the detection results.

[0198] In practical implementation, to ensure consistency of target information in the image and radar point cloud, data synchronization (timestamp alignment) of the two sensors is required before each single-source target detection. This can be achieved using a queue-based approach. After acquiring different sensor data, the image and radar point cloud data are placed into the data queue according to their acquired timestamp order, and then time alignment is performed. The time alignment process includes: storing the radar point cloud data and camera image data into their respective queues, and retrieving the earliest radar data frame F from the radar queue. L0 And the earliest frame image F in the image queue C0 Compare the timestamps of the two, if F C0 The timestamp is less than F L0 timestamp, or F C0 Timestamp and F L0 The difference between timestamps is greater than a preset threshold T diff Then remove F from the queue. C0 Continue to F L0 Compare with the image data of the next frame; when F C0 The timestamp is greater than F L0 And the difference in timestamps is less than the time threshold T. diffIf the data time synchronization is complete, then the threshold T is considered to be in the correct position. diff This is an empirical value, which can generally be set to 0.01 seconds. Since the camera's data acquisition frequency is faster than that of the radar, only the data corresponding to the camera will be deleted here to align with the timestamp.

[0199] In some possible implementations, step S250 above fuses the three-dimensional image detection result and the radar detection result to obtain the target detection result, including:

[0200] The image center point of the target is mapped to the three-dimensional radar coordinate system to obtain the reference projection point of the image center point in the three-dimensional radar coordinate system.

[0201] If the reference projection point is within the radar detection box, then determine the first probability of the radar detection result and the second probability of the 3D image detection result; compare the first probability and the second probability, and take the detection result with the higher probability as the target detection result;

[0202] If the reference projection point is outside the radar detection frame, the 3D image detection result will be used as the target detection result.

[0203] The radar detection results include the radar detection bounding box corresponding to the radar-detected target, and the radar detection bounding box includes the area where the radar point cloud data is located. The first probability and the second probability are obtained when acquiring the radar detection results and the 3D image detection results.

[0204] Specifically, based on the radar detection results, the corresponding radar detection box is determined, and the image center point of the image detection target is mapped to the three-dimensional radar coordinate system to obtain the reference projection point of the image center point in the radar coordinate system. If the reference projection point of the image detection target is within the radar detection box, the first probability of the radar detection result and the second probability of the three-dimensional image detection result are determined. The first probability and the second probability are compared, and the detection result with the higher probability is taken as the target detection result. If the reference projection point is outside the radar detection box, the three-dimensional image detection result is taken as the target detection result.

[0205] In the specific implementation process, the position of the image center point is projected onto the radar coordinate system, along with the corresponding depth information. It is then determined whether the reference projection point of the image center point is within the radar detection box. If the reference projection point of the image center point is within the radar detection box, the probability information of the camera and radar detection results is compared, and the one with the higher probability is selected as the corresponding detection information. The detection category, center point position, length, width, height, azimuth, and other information of the detected target are updated. If the reference projection point is outside the radar detection box, the 3D image detection result corresponding to the camera is taken as the final detection result.

[0206] In the above embodiments, the image detection target is obtained by obtaining a two-dimensional image detection result through camera image data, the depth information of the image detection target in three-dimensional space is calculated based on LiDAR point cloud data, thereby obtaining a three-dimensional image detection result of the image detection target, and the radar detection target is obtained based on the radar detection result of radar point cloud data. The three-dimensional image detection result and the radar detection result are fused to obtain the target detection result. A lightweight detection algorithm is used to obtain detection results in different dimensions, and the two-dimensional image detection result is upgraded based on radar data, thereby fusing the results to obtain the target detection result. This method is suitable for scenarios with low computing power or that do not support complex radar detection, and can obtain accurate target detection results even with limited computing power.

[0207] Furthermore, a three-dimensional unit spherical coordinate system is introduced. When determining the depth information of the image detection target, the radar points corresponding to the radar point cloud data are projected onto the unit spherical coordinate system, and the image center point corresponding to the camera image data is also projected onto the unit spherical coordinate system. In the unit spherical coordinate system, the depth information corresponding to the image detection target is obtained, which can determine the spatial distance between each point and obtain more accurate depth information, making the subsequent detection results more accurate.

[0208] Furthermore, based on the above steps, the 3D image detection results and radar detection results can be obtained, and their consistency can be verified and aligned. Under certain conditions, the two detection results can be fused together, avoiding false detections from a single detection result and missed detections in blind spots of the camera and radar fields of view, thus increasing the comprehensiveness and accuracy of the detection results.

[0209] In one example, the detection method of this application that combines lidar and camera, such as... Figure 9 As shown, it may include:

[0210] Acquire radar point cloud data and camera image data, along with their respective acquisition timestamps;

[0211] Based on the acquired timestamps, the radar point cloud data is aligned with the timestamps of the camera image data;

[0212] Target detection is performed on camera image data to obtain two-dimensional image detection results (i.e., 2D target detection as shown in the figure);

[0213] Determine the image center point of the target in the two-dimensional image detection result; project the image center point from the two-dimensional pixel coordinate system to the unit spherical coordinate system (i.e., the 2D back projection shown in the figure) to obtain the image projection point of the image center point in the unit spherical coordinate system;

[0214] Based on the preset radar resolution, the radar point cloud data is divided into multiple voxel grids, and the voxel coordinates of each voxel grid are determined. The voxel coordinates are then projected from the three-dimensional radar coordinate system to the two-dimensional pixel coordinate system to obtain the radar pixel points of the voxel coordinates in the two-dimensional pixel coordinate system.

[0215] Determine the pixel distance between each radar pixel and the image center point; identify radar pixels with a pixel distance less than a preset distance threshold as target radar pixels; determine the target voxel grid corresponding to the target radar pixels; project the target voxel coordinates of the target voxel grid from the 3D radar coordinate system to the unit spherical coordinate system to obtain the radar projection points of the target voxel coordinates in the unit spherical coordinate system; based on the image projection points and radar projection points, determine the depth information corresponding to the image detection target (i.e., depth recovery as shown in the figure) to obtain the 3D image detection result;

[0216] The radar point cloud data is preprocessed, and the radar detection results (i.e., 3D target detection as shown in the figure) are obtained based on the radar point cloud data.

[0217] The image center point of the detected target is mapped to a 3D radar coordinate system to obtain a reference projection point of the image center point in the 3D radar coordinate system. Based on the radar detection results, the corresponding radar detection box is determined. If the reference projection point is within the radar detection box, the first probability of the radar detection result and the second probability of the 3D image detection result are determined. The first probability and the second probability are compared, and the detection result with the higher probability is taken as the target detection result. If the reference projection point is outside the radar detection box, the 3D image detection result is taken as the target detection result (i.e., consistency check and information fusion as shown in the figure).

[0218] Determine the target detection results and output them (i.e., the output fusion result shown in the figure).

[0219] The aforementioned detection method combining LiDAR and camera obtains a two-dimensional image detection result of the target from camera image data, calculates the depth information of the target in three-dimensional space based on LiDAR point cloud data, thus obtaining a three-dimensional image detection result of the target, and obtains a radar detection result based on the radar detection result of the radar point cloud data. The three-dimensional image detection result and the radar detection result are then fused to obtain the target detection result. A lightweight detection algorithm is used to obtain detection results in different dimensions, and the two-dimensional image detection result is upgraded based on radar data, thus fusing the results to obtain the target detection result. This method is suitable for scenarios with low computing power or that do not support complex radar detection, and can obtain accurate target detection results even with limited computing power.

[0220] Furthermore, a three-dimensional unit spherical coordinate system is introduced. When determining the depth information of the image detection target, the radar points corresponding to the radar point cloud data are projected onto the unit spherical coordinate system, and the image center point corresponding to the camera image data is also projected onto the unit spherical coordinate system. In the unit spherical coordinate system, the depth information corresponding to the image detection target is obtained, which can determine the spatial distance between each point and obtain more accurate depth information, making the subsequent detection results more accurate.

[0221] Furthermore, based on the above steps, the 3D image detection results and radar detection results can be obtained, and their consistency can be verified and aligned. Under certain conditions, the two detection results can be fused together, avoiding false detections from a single detection result and missed detections in blind spots of the camera and radar fields of view, thus increasing the comprehensiveness and accuracy of the detection results.

[0222] This application provides a detection device that fuses lidar and camera, such as... Figure 10 As shown, the detection device 1000 that fuses lidar and camera may include: a data acquisition module 1001, an image detection module 1002, a depth acquisition module 1003, a lidar detection module 1004, and a result fusion module 1005, wherein,

[0223] The data acquisition module 1001 is used to acquire timestamp-aligned camera image data and radar point cloud data;

[0224] Image detection module 1002 is used to obtain two-dimensional image detection results based on camera image data; the two-dimensional image detection results include the image detection target;

[0225] The depth acquisition module 1003 is used to determine the depth information corresponding to the image detection target based on radar point cloud data, and obtain the three-dimensional image detection result;

[0226] The radar detection module 1004 is used to obtain radar detection results based on radar point cloud data; the radar detection results include radar detected targets.

[0227] The result fusion module 1005 is used to fuse the three-dimensional image detection results and the radar detection results to obtain the target detection results.

[0228] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0229] Determine the center point of the target image;

[0230] Based on radar point cloud data and image center points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0231] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0232] Based on the preset radar resolution, the radar point cloud data is divided into multiple voxel grids;

[0233] Determine the voxel coordinates of each voxel mesh;

[0234] Based on the image center point and voxel coordinates, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0235] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0236] Projecting the image center point from the two-dimensional pixel coordinate system to the unit sphere coordinate system yields the image projection point of the image center point in the unit sphere coordinate system. The unit sphere coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin, and includes a unit sphere centered on the camera optical center.

[0237] Project each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system.

[0238] Based on image projection points and radar projection points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0239] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0240] Project each voxel coordinate point from the three-dimensional radar coordinate system to the two-dimensional pixel coordinate system to obtain the radar pixel point of the voxel coordinate point in the two-dimensional pixel coordinate system.

[0241] Determine the pixel distance between each radar pixel and the image center point;

[0242] Radar pixels whose pixel distance is less than a preset distance threshold are designated as target radar pixels.

[0243] Determine the target voxel grid corresponding to the target radar pixel;

[0244] Projecting the target voxel coordinates from the three-dimensional radar coordinate system to the unit spherical coordinate system yields the radar projection points of the target voxel coordinates in the unit spherical coordinate system.

[0245] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0246] Determine the spatial distance between each radar projection point and the image projection point in the unit spherical coordinate system;

[0247] Based on spatial distance, a preset number of candidate radar points are selected from multiple radar projection points;

[0248] Based on preset screening criteria, at least three target radar points are selected from the candidate radar points;

[0249] Based on at least three target radar points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0250] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0251] Based on spatial distance, candidate radar points are arranged from smallest to largest;

[0252] Multiple candidate groups are obtained by dividing the radar points in the order of arrangement; each candidate group includes at least three candidate radar points.

[0253] Determine the distance between any two candidate radar points in the candidate group in turn;

[0254] Candidate groups whose distances meet the preset screening criteria are selected as target radar groups;

[0255] The candidate radar points in the target radar group are used as the target radar points.

[0256] As an optional embodiment, in this device, the depth acquisition module 1003 is specifically used for:

[0257] Determine the target plane based on at least three target radar points;

[0258] Based on the distance between the target plane and the origin of the unit spherical coordinate system, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

[0259] As an optional embodiment, in this device, the data acquisition module 1001 is specifically used for:

[0260] Obtain camera image data and corresponding image timestamps;

[0261] Acquire radar point cloud data and the corresponding radar timestamps;

[0262] Align the image timestamps and radar timestamps to obtain timestamp-aligned camera image data and radar point cloud data.

[0263] As an optional embodiment, in this device, the result fusion module 1005 is specifically used for:

[0264] The image center point of the target is mapped to the three-dimensional radar coordinate system to obtain the reference projection point of the image center point in the three-dimensional radar coordinate system; the radar detection result includes the radar detection box corresponding to the target.

[0265] If the reference projection point is within the radar detection box, then determine the first probability of the radar detection result and the second probability of the 3D image detection result; compare the first probability and the second probability, and take the detection result with the higher probability as the target detection result;

[0266] If the reference projection point is outside the radar detection frame, the 3D image detection result will be used as the target detection result.

[0267] The detection device that integrates LiDAR and camera provided in this application obtains a two-dimensional image detection result of the target through camera image data, calculates the depth information of the target in three-dimensional space based on LiDAR point cloud data, thereby obtaining a three-dimensional image detection result of the target, and obtains a radar detection target based on the radar detection result of the radar point cloud data. The three-dimensional image detection result and the radar detection result are then fused to obtain the target detection result. A lightweight detection algorithm is used to obtain detection results in different dimensions, and the two-dimensional image detection result is upgraded based on radar data, thereby fusing the results to obtain the target detection result. This device is suitable for scenarios with low computing power or that do not support complex radar detection, and can obtain accurate target detection results even with limited computing power.

[0268] Furthermore, a three-dimensional unit spherical coordinate system is introduced. When determining the depth information of the image detection target, the radar points corresponding to the radar point cloud data are projected onto the unit spherical coordinate system, and the image center point corresponding to the camera image data is also projected onto the unit spherical coordinate system. In the unit spherical coordinate system, the depth information corresponding to the image detection target is obtained, which can determine the spatial distance between each point and obtain more accurate depth information, making the subsequent detection results more accurate.

[0269] Furthermore, based on the above steps, the 3D image detection results and radar detection results can be obtained, and their consistency can be verified and aligned. Under certain conditions, the two detection results can be fused together, avoiding false detections from a single detection result and missed detections in blind spots of the camera and radar fields of view, thus increasing the comprehensiveness and accuracy of the detection results.

[0270] The apparatus in this application embodiment can execute the method provided in this application embodiment, and its implementation principle is similar, and it has corresponding technical effects. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For a detailed functional description of each module of the apparatus, please refer to the description in the corresponding method shown above, which will not be repeated here.

[0271] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of this application. Compared with the prior art, it can achieve: obtaining a two-dimensional image detection result of the image detection target through camera image data, calculating the depth information of the image detection target in three-dimensional space based on LiDAR point cloud data to obtain a three-dimensional image detection result of the image detection target, and obtaining a radar detection target based on the radar detection result of the radar point cloud data. The three-dimensional image detection result and the radar detection result are fused to obtain the target detection result. A lightweight detection algorithm is used to obtain detection results of different dimensions, and the two-dimensional image detection result is upgraded based on radar data, thereby fusing the results to obtain the target detection result. It is suitable for scenarios with low computing power, such as those that do not support complex radar detection, and can obtain accurate target detection results even with limited computing power.

[0272] In one alternative embodiment, an electronic device is provided, such as Figure 11 As shown, Figure 11 The illustrated electronic device 11000 includes a processor 11001 and a memory 11003. The processor 11001 and the memory 11003 are connected, for example, via a bus 11002. Optionally, the electronic device 11000 may further include a transceiver 11004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 11004 is not limited to one type, and the structure of the electronic device 11000 does not constitute a limitation on the embodiments of this application.

[0273] Processor 11001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 11001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0274] Bus 11002 may include a pathway for transmitting information between the aforementioned components. Bus 11002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 11002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0275] The memory 11003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0276] The memory 11003 is used to store computer programs that execute the embodiments of this application, and the execution is controlled by the processor 11001. The processor 11001 is used to execute the computer programs stored in the memory 11003 to implement the steps shown in the foregoing method embodiments.

[0277] Among them, electronic devices include, but are not limited to: devices that can realize the above-mentioned detection method of fusion of lidar and camera in industrial or embedded scenarios.

[0278] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0279] It should be noted that the computer-readable storage medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0280] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0281] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0282] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0283] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0284] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A detection method that fuses lidar and camera, characterized in that, include: Acquire timestamp-aligned camera image data and radar point cloud data; The scene corresponding to the camera image data and the radar point cloud data is an industrial robot development scenario. Based on the camera image data, a two-dimensional image detection result is obtained; the two-dimensional image detection result includes the image detection target; Based on the radar point cloud data, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained; Based on the radar point cloud data, the radar detection results are obtained; The radar detection results include radar-detected targets; The target detection result is obtained by fusing the three-dimensional image detection result and the radar detection result; The process of determining the depth information corresponding to the image detection target based on the radar point cloud data to obtain a three-dimensional image detection result includes: The image center point of the target to be detected is determined; the image center point is projected from the two-dimensional pixel coordinate system to the unit sphere coordinate system to obtain the image projection point of the image center point in the unit sphere coordinate system; the unit sphere coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin, and the unit sphere coordinate system includes a unit sphere centered on the camera optical center; Based on a preset radar resolution, the radar point cloud data is divided into multiple voxel grids; the voxel coordinate points of each voxel grid are determined; and each voxel coordinate point is projected from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system. Based on the image projection points and the radar projection points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained; The process of fusing the 3D image detection result and the radar detection result to obtain the target detection result includes: Extract features from the 3D image detection results and extract features from the radar detection results; The target detection result is obtained by associating the features of the radar detection result with the features of the three-dimensional image detection result through a feature matching algorithm.

2. The detection method for fusion of lidar and camera according to claim 1, characterized in that, The step of projecting each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system includes: Project each voxel coordinate point from the three-dimensional radar coordinate system to the two-dimensional pixel coordinate system to obtain the radar pixel point of the voxel coordinate point in the two-dimensional pixel coordinate system. Determine the pixel distance between each radar pixel and the center point of the image; Radar pixels whose pixel distance is less than a preset distance threshold are taken as target radar pixels; Determine the target voxel grid corresponding to the target radar pixel; The step of projecting each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system includes: The target voxel coordinates of the target voxel mesh are projected from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection points of the target voxel coordinates in the unit spherical coordinate system.

3. The detection method for fusion of lidar and camera according to claim 1, characterized in that, The step of determining the depth information corresponding to the image detection target based on the image projection point and the radar projection point to obtain the three-dimensional image detection result includes: Determine the spatial distance between each radar projection point and the image projection point in the unit spherical coordinate system; Based on the spatial distance, a preset number of candidate radar points are selected from the plurality of radar projection points; Based on preset screening criteria, at least three target radar points are selected from the candidate radar points; Based on the at least three target radar points, the depth information corresponding to the image detection target is determined, and a three-dimensional image detection result is obtained.

4. The detection method for fusion of lidar and camera according to claim 3, characterized in that, The step of selecting at least three target radar points from the candidate radar points based on preset screening conditions includes: Based on the spatial distance, the candidate radar points are arranged from smallest to largest; Multiple candidate groups are obtained by sequentially dividing the data according to the aforementioned arrangement order; each candidate group includes at least three candidate radar points. The distance between any two candidate radar points in the candidate group is determined sequentially. Candidate groups whose distances meet preset screening conditions are selected as target radar groups; The candidate radar points in the target radar group are used as the target radar points.

5. The detection method for fusion of lidar and camera according to claim 3, characterized in that, The process of determining the depth information corresponding to the image detection target based on the at least three target radar points to obtain a three-dimensional image detection result includes: Based on the at least three target radar points, the target plane is determined; Based on the distance between the target plane and the origin of the unit spherical coordinate system, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained.

6. The detection method for fusion of lidar and camera according to claim 1, characterized in that, The acquisition of timestamp-aligned camera image data and radar point cloud data includes: Obtain camera image data and corresponding image timestamps; Acquire radar point cloud data and the corresponding radar timestamps; Align the image timestamp and the radar timestamp to obtain the timestamp-aligned camera image data and radar point cloud data.

7. The detection method for fusion of lidar and camera according to claim 1, characterized in that, The radar detection results include the radar detection bounding boxes corresponding to the radar-detected targets; The process of fusing the 3D image detection result and the radar detection result to obtain the target detection result includes: The image center point of the target detected by the image is mapped to the three-dimensional radar coordinate system to obtain the reference projection point of the image center point in the three-dimensional radar coordinate system; If the reference projection point is within the radar detection frame, then a first probability of the radar detection result and a second probability of the three-dimensional image detection result are determined; the first probability and the second probability are compared, and the detection result with the larger probability is taken as the target detection result; If the reference projection point is outside the radar detection frame, then the three-dimensional image detection result is used as the target detection result.

8. A detection device that fuses lidar and camera, characterized in that, include: The data acquisition module is used to acquire timestamp-aligned camera image data and radar point cloud data; The scene corresponding to the camera image data and the radar point cloud data is an industrial robot development scenario. An image detection module is used to obtain a two-dimensional image detection result based on the camera image data; the two-dimensional image detection result includes the image detection target; The depth acquisition module is used to determine the depth information corresponding to the image detection target based on the radar point cloud data, and obtain the three-dimensional image detection result; A radar detection module is used to obtain radar detection results based on the radar point cloud data; The radar detection results include radar-detected targets; The result fusion module is used to fuse the three-dimensional image detection results and the radar detection results to obtain the target detection results; The depth acquisition module is specifically used for: The image center point of the target to be detected is determined; the image center point is projected from the two-dimensional pixel coordinate system to the unit sphere coordinate system to obtain the image projection point of the image center point in the unit sphere coordinate system; the unit sphere coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin, and the unit sphere coordinate system includes a unit sphere centered on the camera optical center; Based on a preset radar resolution, the radar point cloud data is divided into multiple voxel grids; the voxel coordinates of each voxel grid are determined. Project each voxel coordinate point from the three-dimensional radar coordinate system to the unit spherical coordinate system to obtain the radar projection point of the voxel coordinate point in the unit spherical coordinate system. Based on the image projection points and the radar projection points, the depth information corresponding to the image detection target is determined, and the three-dimensional image detection result is obtained; The result fusion module is specifically used for: Extract features from the 3D image detection results and extract features from the radar detection results; The target detection result is obtained by associating the features of the radar detection result with the features of the three-dimensional image detection result through a feature matching algorithm.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the detection method for fusion of lidar and camera as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the detection method of fusion of lidar and camera as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the detection method of fusion of lidar and camera as described in any one of claims 1-7.

Images