Target localization method based on multi-resolution asymmetric fusion of laser point cloud and 2D image

Abstract

The present application provides a target positioning method for multi-resolution asymmetric fusion of laser point cloud and two-dimensional image, which includes the following steps: collecting laser point cloud and two-dimensional image; inputting the two-dimensional image into a pre-built target detection network model and outputting target detection results; wherein the target detection results include the target area; performing field-of-view alignment on the laser point cloud and the two-dimensional image; projecting the laser point cloud into a multi-resolution depth map, performing pixel-level alignment on the depth map with the pyramid features of the two-dimensional image, and obtaining array data for each pixel of the depth map after pixel-level alignment; and calculating the distance value between the target and the laser radar based on the array data of each pixel of the depth map within the target area. The present application realizes the association between laser point cloud data and target detection results through the multi-resolution depth map converted from the laser point cloud, obtains the distance of each detected target, improves target positioning accuracy, and increases target detection speed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a target positioning method using multi-resolution asymmetric fusion of laser point clouds and two-dimensional images. Background Art

[0002] Currently, image-based two-dimensional object detection is an extremely important application of artificial intelligence. It has the ability to simultaneously identify the categories and two-dimensional positions of multiple objects in a single image frame. It is widely used in many fields such as autonomous driving, intelligent security, and industrial manufacturing inspection. Currently, the single-stage object detection network model based on convolutional neural networks (CNN) has become the mainstream method in the industry. The single-stage object detection network adopts a feature pyramid architecture. That is, after the image passes through the object detection backbone network, multiple feature layers with different resolutions are formed, and object detection is performed based on the features of each layer.

[0003] However, when two-dimensional target detection in the camera projection space is directly applied to the field of autonomous driving, it faces the problem of being unable to accurately estimate the distance of the target from the camera, and therefore unable to locate the target, which in turn makes it difficult for autonomous vehicles to perform effective path planning and dynamic obstacle avoidance.

[0004] Currently, lidar (LiDAR) has the capabilities of three-dimensional ranging and point cloud scanning, capable of determining a target's distance, orientation, and altitude relative to the LiDAR. It has become a crucial sensor in autonomous vehicles and is typically used for three-dimensional target detection—determining the target's category and rectangular box. The rectangular box contains the target's distance relative to the LiDAR, its orientation, and its dimensions (length, width, and height). However, LiDAR has its limitations. The laser point cloud is sparse and lacks essential texture information. Furthermore, extracting features from this point cloud data requires a large amount of computational resources and storage space for a 3D CNN.

[0005] Tesla, a major player in the autonomous driving field, has proposed a system to reduce the overall cost of autonomous driving systems. This system uses a multi-camera surround view system and a Transformer architecture to detect object categories and their positions within the Bird's Eye View (BEV) space. This information, relative to the camera's projection space, supports effective path planning and obstacle avoidance for autonomous vehicles.

[0006] With the mass production of LiDAR, LiDAR prices are rapidly declining. Compared to using low-cost cameras alone, the multimodal fusion of LiDAR and cameras is gaining increasing attention from academia and industry. The fusion of LiDAR and cameras is generally divided into fusion at the raw data level, such as laser point clouds and image pixels (also known as front fusion or data-level fusion), fusion at the feature level (also known as deep fusion or feature-level fusion), and fusion of target detection results (also known as back fusion or target-level fusion). As research deepens, fusion of different modalities and levels has also emerged, such as the fusion of raw laser point cloud data with CNN features of two-dimensional images, generally referred to as asymmetric fusion. However, current multimodal fusion methods for laser point clouds and two-dimensional images rely on three-dimensional annotation, that is, labeling objects in the scene with rectangular boxes. This is more time-consuming than traditional two-dimensional rectangular box annotation, significantly increases the workload of manual annotation, and makes it difficult to reduce costs, quickly expand training sets, or leverage existing rich open training sets for two-dimensional images to improve target detection performance.

[0007] For applications like border inspections, where targets are sparsely distributed, mutual occlusion and clustering between targets are rare. Directly adopting the multimodal fusion 3D target detection method from the autonomous driving field using laser point clouds and 2D images would result in numerous issues, including high algorithm complexity, high computing power requirements, and limited dataset expansion capabilities.

[0008] Considering that two-dimensional target detection in the camera's projection space provides azimuth information, namely the left and right deviation angle of the target relative to the camera's central axis, the laser point cloud data output by the lidar can be used to associate the point cloud with the target through azimuth to determine the target's distance from the camera, and thus the target's position in the BEV space. However, directly applying this simple fusion algorithm faces many problems, such as the non-unique association between the laser point cloud and the two-dimensional target when a small target partially occludes a larger target, and the presence of measurement noise in the laser point cloud and noise caused by the climatic environment, which can easily lead to false detections. These problems can lead to inaccurate target positioning or false detections.

[0009] Therefore, the technical problems that need to be solved urgently are: how to improve the accuracy of target positioning and how to increase the speed of target positioning detection. Summary of the Invention

[0010] The purpose of this application is to provide a multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image, which is aimed at sparse target outdoor autonomous driving scenarios. It uses a single-stage two-dimensional target detection network model based on a feature pyramid, and converts the original laser point cloud data into a three-dimensional point cloud and a multi-resolution two-dimensional depth map according to the downsampling rate parameter of the feature pyramid. The laser point cloud data and target detection results are associated through the multi-resolution depth map to obtain the distance of each detected target, thereby improving the target positioning accuracy and the target detection speed.

[0011] To achieve the above-mentioned purpose, the present application provides a multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image, characterized in that the method includes the following steps: collecting laser point cloud and two-dimensional image; inputting the two-dimensional image into a pre-built target detection network model, and outputting the target detection result; wherein the target detection result includes the target area; performing field-of-view level alignment on the laser point cloud and the two-dimensional image; projecting the laser point cloud into a multi-resolution depth map, performing pixel-level alignment on the depth map and the pyramid features of the two-dimensional image, and obtaining array data of each pixel of the depth map after pixel-level alignment; and calculating the distance value between the target and the laser radar based on the array data of each pixel of the depth map in the target area.

[0012] The multi-resolution asymmetric fusion target positioning method of the laser point cloud and the two-dimensional image as described above, wherein, before calculating the distance value between the target and the laser radar, it also includes: filtering out falsely detected targets based on the array data of each pixel of the depth map in the target area.

[0013] The multi-resolution asymmetric fusion target positioning method of the laser point cloud and the two-dimensional image as described above, wherein the array data of the pixel includes the distance average, the distance variance, the average reflection intensity and the number of point clouds.

[0014] The multi-resolution asymmetric fusion target positioning method of the laser point cloud and the two-dimensional image as described above, wherein the method for calculating the distance value between the target and the laser radar based on the array data of each pixel of the depth map within the target area includes: calculating the distance value between the target and the laser radar based on the distance variance and average reflection intensity of each pixel of the depth map within the target area.

[0015] The multi-resolution asymmetric fusion target positioning method of the laser point cloud and the two-dimensional image as described above, wherein the method of filtering the falsely detected targets according to the array data of each pixel of the depth map within the target area includes: filtering the falsely detected targets according to the number of point clouds in the array data of each pixel of the depth map within the target area.

[0016] As described above, the multi-resolution asymmetric fusion target positioning method of the laser point cloud and the two-dimensional image, wherein the method for field-of-view level alignment of the laser point cloud and the two-dimensional image includes the following steps: filtering the laser point cloud located outside the horizontal field of view angle of the camera; based on the target detection result, filtering targets whose target areas are all outside the vertical field of view angle of the laser radar.

[0017] The multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image as described above, wherein the method for filtering falsely detected targets according to the number of point clouds in the array data of each pixel of the depth map in the target area includes the following sub-steps: selecting a depth map of corresponding resolution for each target according to the pyramid feature layer corresponding to the target in the target detection result; wherein different pyramid feature layers correspond to depth maps of different resolutions; according to the target area in the target detection result, counting the number of pixels in the depth map within the target area with a point cloud number of 0; and filtering falsely detected targets according to the counted number of pixels.

[0018] As described above, in the multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image, if the number of pixels with a point cloud quantity of 0 in the depth map within the target frame is greater than a specified threshold, the falsely detected targets are filtered out; otherwise, there is no need to filter the falsely detected targets.

[0019] The multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image as described above, wherein the laser point cloud is collected by a laser radar; and the two-dimensional image is collected by a camera.

[0020] The multi-resolution asymmetric fusion target positioning method of the laser point cloud and the two-dimensional image as described above further includes: performing target positioning from a bird's-eye view according to the distance value between the target and the laser radar and the azimuth angle of the target in the two-dimensional image.

[0021] The beneficial effects achieved by this application are as follows:

[0022] (1) Compared with the target positioning method in the prior art that uses multimodal fusion of laser point cloud and two-dimensional image based on three-dimensional rectangular box annotation information, the present application directly uses the original data of laser point cloud, without the need to use high-computation three-dimensional CNN to extract point cloud features, which significantly reduces the computing power requirements of the entire network model and improves the target positioning detection speed.

[0023] (2) After aligning the laser point cloud and the two-dimensional image at the field of view level, the present application projects the laser point cloud into a multi-resolution two-dimensional depth map, and filters out falsely detected targets caused by environmental noise in the laser point cloud based on the number of point clouds contained in the depth map within the target area; then, based on the variance of the reflection intensity and distance of each pixel in the depth map within the target area, it is ensured that when a small target partially blocks a large target, the laser point cloud can be associated with the small target and the large target respectively, so as to obtain accurate estimates of the laser radar at different target distances and improve the target positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can also be obtained based on these drawings.

[0025] Figure 1 This is a flow chart of a target positioning method using multi-resolution asymmetric fusion of laser point cloud and two-dimensional image according to an embodiment of the present application.

[0026] Figure 2 This is a flow chart of a method for field-of-view level alignment of a laser point cloud and a two-dimensional image according to an embodiment of the present application.

[0027] Figure 3 This is a flow chart of a method for filtering misdetected targets according to the number of point clouds in array data of each pixel in a depth map within a target area according to an embodiment of the present application.

[0028] Figure 4 This is a structural diagram of a multi-resolution asymmetric fusion target positioning system of a laser point cloud and a two-dimensional image in an embodiment of the present application.

[0029] Figure numerals: 10 - acquisition module; 20 - target detection module; 30 - field-of-view level alignment module; 40 - acquisition module; 50 - filtering module; 60 - calculation module; 70 - positioning module; 100 - target positioning system. DETAILED DESCRIPTION

[0030] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0031] Example 1

[0032] like Figure 1 As shown, the present application provides a multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image, which includes the following steps:

[0033] Step S1, collecting laser point clouds and two-dimensional images.

[0034] As a specific embodiment of the present invention, in an autonomous driving scenario, laser point clouds are collected by a lidar; and two-dimensional images are collected by a camera.

[0035] Step S2: Input the two-dimensional image into the pre-built target detection network model and output the target detection result.

[0036] As a specific embodiment of the present invention, the two-dimensional image is a two-dimensional image captured by a camera for a sparsely targeted outdoor autonomous driving scene, and the two-dimensional image contains a target to be detected.

[0037] As a specific embodiment of the present invention, the pre-built object detection network model is a single-stage two-dimensional object detection network model based on a feature pyramid, such as the YOLO (You Only Look Once) series of models. A multi-resolution object detection method built into the feature pyramid.

[0038] In one embodiment of the present invention, the object detection result includes the object category, confidence level, object region (the location of the object in a two-dimensional image, often represented by a rectangular border), and the number of the pyramid feature layer where the object is located. The pyramid feature layer is a layer of a single-stage two-dimensional object detection network model based on a feature pyramid.

[0039] Taking YOLO as an example, the pyramid feature layers are numbered 3, 4, or 5, corresponding to resolutions of 1 / 8, 1 / 16, and 1 / 32 of the input 2D image, respectively. The pyramid feature layer where the target is located can also be numbered 3, 4, or 5, etc. Different pyramid feature layers correspond to depth maps of different resolutions.

[0040] Step S3: perform field-of-view level alignment on the laser point cloud and the two-dimensional image.

[0041] like Figure 2 As shown, step S3 includes the following sub-steps:

[0042] Step S310: filtering the laser point cloud outside the horizontal field of view of the camera.

[0043] Specifically, according to the external and internal parameters of the lidar and camera and the pinhole imaging model, the laser point cloud located outside the horizontal field of view (FoV) of the camera is filtered (the horizontal field of view of the lidar is usually larger than that of a single camera).

[0044] Step S320: Filter targets whose target areas (frames) are all outside the vertical field of view of the laser radar according to the target detection results.

[0045] Specifically, based on the external and internal parameters of the lidar and camera and the pinhole imaging model, targets whose frames are all outside the vertical field of view of the lidar (the vertical field of view of the camera is usually larger than that of the lidar) are filtered out, and the distances of these targets are set to Unknown to achieve field of view alignment between the laser point cloud and the two-dimensional image.

[0046] Step S4: Project the laser point cloud into a multi-resolution depth map, perform pixel-level alignment on the depth map with the pyramid features of the two-dimensional image, and obtain array data of each pixel of the depth map after pixel-level alignment.

[0047] As a specific embodiment of the present invention, after the laser point cloud and the two-dimensional image are aligned at the field of view level, the laser point cloud within the horizontal field of view of the camera is projected from the three-dimensional space to the projection space, and multiple depth maps of different resolutions are formed according to the resolution of the two-dimensional image corresponding to the pyramid feature layer. For example, the laser point cloud is projected into a two-dimensional depth map with a resolution of 1 / 8, 1 / 16, and 1 / 32 of the input two-dimensional image resolution. Pixel-level alignment of the pyramid feature layer and the depth map is achieved, or in other words, pixel-level alignment of the multi-resolution depth map projected by the laser point cloud with the pyramid features of the two-dimensional image is performed.

[0048] As a specific embodiment of the present invention, the array data of pixels includes distance average, distance variance, average reflection intensity and point cloud quantity.

[0049] It should be explained that the average distance refers to the average distance between the pixel and the lidar; the distance variance refers to the distance variance between the pixel and the lidar; the average reflection intensity refers to the average reflection intensity between the pixel and the lidar; and the number of point clouds refers to the number of point clouds falling within the pixel.

[0050] Because lidar has smaller horizontal and vertical resolutions than cameras, taking a high-definition camera with a 48° horizontal field of view, a 30° vertical field of view, and a 1920×1080 resolution as an example, the horizontal resolution of a single pixel is 48 / 1920 = 0.025°, and the vertical resolution is 30 / 1080 = 0.028°. In contrast, lidar has a horizontal resolution between 0.09 and 0.2°, a vertical resolution of the RoI (RoI) region of interest between 0.1 and 0.3°, and even exceeds 0.5° in non-RoI regions. After pixel-level alignment of the multi-resolution depth map projected by the laser point cloud with the pyramid features of the 2D image, each pixel in the depth map at different resolutions may contain zero or multiple point clouds. To preserve this point cloud information, each pixel is represented by an array. The first value in the array represents the average distance between the pixel and the lidar, the second value represents the average reflection intensity between the pixel and the lidar, the third value represents the variance of the distance between the pixel and the lidar, and the fourth value represents the number of point clouds within the pixel. If there is no point cloud in the pixel, the first, second, and third values ​​in the array are marked as Unknown (-1), and the fourth value is 0.

[0051] Step S5: filtering out falsely detected targets based on the array data of each pixel in the depth map within the target area.

[0052] As a specific embodiment of the present invention, filtering the falsely detected targets according to the array data of each pixel of the depth map within the target area includes: filtering the falsely detected targets according to the number of point clouds in the array data of each pixel of the depth map within the target area.

[0053] As a specific embodiment of the present invention, based on the target area in the target detection result and the array data of each pixel of the depth map in the target area, the number of point clouds corresponding to each pixel in the depth map in the target area is obtained, and the false detection targets are filtered according to the number of point clouds.

[0054] As a specific embodiment of the present invention, the number of pixels with zero point clouds in the depth map of the resolution corresponding to the pyramid feature layer of the target is obtained according to the number of point clouds in the array data of each pixel of the depth map within the target area; based on the number of pixels with zero point clouds in the depth map of the resolution corresponding to the pyramid feature layer of the target, falsely detected targets caused by point cloud noise or noise are filtered out.

[0055] like Figure 3 As shown, the method for filtering misdetected targets according to the number of point clouds in the array data of each pixel of the depth map within the target area includes the following sub-steps:

[0056] Step S510 : selecting a depth map of corresponding resolution for each target according to the pyramid feature layer corresponding to the target in the target detection result.

[0057] Specifically, according to the pyramid feature layer corresponding to the target in the target detection result output by the target detection network model, a depth map of corresponding resolution is selected for each target T0.

[0058] Step S520 : According to the target area in the target detection result, the number of pixels in the depth map within the target area whose point cloud quantity is 0 is counted.

[0059] Specifically, the target area is the border of the target T0, and the number of pixels with a point cloud quantity of 0 in the depth map within the target border (that is, the depth map of the corresponding resolution is selected for the target T0) is counted, which is recorded as N.

[0060] Step S530: Filtering misdetected objects based on the counted number of pixels.

[0061] Specifically, if the number of pixels N with a point cloud count of 0 in the depth map within the target frame is greater than the specified threshold Nt, the target is considered to be a false target or a misdetected target, and the target is directly filtered out to eliminate false detection and incorrect positioning of the target caused by measurement noise and environmental noise in the laser point cloud. Otherwise, there is no need to filter the target.

[0062] As an optional embodiment of the present invention, the confidence level of the target may be lowered, and the distance value of the target may be marked as Unknown.

[0063] Step S6, calculating the distance between the target and the laser radar based on the array data of each pixel in the depth map within the target area.

[0064] As a specific embodiment of the present invention, based on the average reflection intensity and distance variance corresponding to each pixel in the depth map within the target area, the distance values ​​between the remaining targets and the laser radar after filtering out the falsely detected targets are calculated.

[0065] Step S6 includes the following sub-steps:

[0066] Step S610 , calculating the distance variance and reflection intensity variance of all pixels within the frame corresponding to the target in the depth map.

[0067] Because the reflection intensities of different targets vary, and in sparse target scenarios, the distances from the target to the LiDAR also vary significantly, we calculate the distance variance Vd and reflection intensity variance Vi for all pixels within the bounding box corresponding to target T0 in the depth map. The reflection intensity variance Vi is calculated based on the average reflection intensity corresponding to each pixel in the depth map within the target area. This calculation method uses existing methods and will not be repeated here.

[0068] In step S620, the distance variance and the reflection intensity variance of all pixels in the frame corresponding to the target in the depth map are compared with the first threshold and the second threshold, and the distance value between the target and the laser radar is calculated based on the comparison results.

[0069] The first comparison result is that if any of the distance variance Vd and reflection intensity variance Vi of all pixels in the target corresponding frame in the depth map is greater than the corresponding specified first threshold and the second threshold This indicates that the corresponding target border may contain distance information of other targets. Therefore, the distance between each pixel of the depth map within the target border and the lidar is sorted, and the minimum distance and the maximum distance are equally divided into L distance intervals. The number of pixels falling into each distance interval is counted, and the middle value of the distance interval with the largest number of pixels is taken as the distance value between the target and the lidar, thereby solving the problem of non-unique association between the laser point cloud and the two-dimensional target when a small target partially blocks a large target.

[0070] The second comparison result is that if none of the distance variance Vd and reflection intensity variance Vi of all pixels in the target corresponding frame in the depth map is greater than the corresponding specified first threshold and the second threshold The average distance between all pixels in the depth map within the target frame and the lidar is taken as the distance value between the target and the lidar.

[0071] Step S7: locating the target from a bird's-eye view according to the distance between the target and the laser radar and the azimuth of the target in the two-dimensional image.

[0072] Specifically, based on the distance value between the target and the lidar, its position in the two-dimensional image (i.e., the azimuth angle) is determined or estimated, and then based on the distance value between the target and the lidar and the azimuth angle of the target in the two-dimensional image, the position of the target in the BEV (bird's-eye view) space is obtained, completing the target positioning under the bird's-eye view.

[0073] Example 2

[0074] like Figure 4 As shown, the present application also provides a multi-resolution asymmetric fusion target positioning system 100 of laser point cloud and two-dimensional image, which includes:

[0075] The acquisition module 10 is used to acquire laser point clouds and two-dimensional images.

[0076] The target detection module 20 is used to input the two-dimensional image into a pre-built target detection network model and output the target detection result.

[0077] The target detection result includes the target area.

[0078] The field-of-view level alignment module 30 is used to perform field-of-view level alignment on the laser point cloud and the two-dimensional image.

[0079] The acquisition module 40 is used to project the laser point cloud into a multi-resolution depth map, perform pixel-level alignment on the depth map with the pyramid features of the two-dimensional image, and obtain array data of each pixel of the depth map after pixel-level alignment.

[0080] The filtering module 50 is configured to filter out falsely detected targets based on the array data of each pixel in the depth map within the target area.

[0081] The calculation module 60 is used to calculate the distance value between the target and the laser radar based on the array data of each pixel in the depth map within the target area.

[0082] The positioning module 70 is used to locate the target from a bird's-eye view according to the distance between the target and the laser radar and the azimuth of the target in the two-dimensional image.

[0083] The beneficial effects achieved by this application are as follows:

[0084] (1) Compared with the target positioning method in the prior art that uses multimodal fusion of laser point cloud and two-dimensional image based on three-dimensional rectangular box annotation information, the present application directly uses the original data of laser point cloud, without the need to use high-computation three-dimensional CNN to extract point cloud features, which significantly reduces the computing power requirements of the entire network model and improves the target positioning detection speed.

[0085] (2) After aligning the laser point cloud and the two-dimensional image at the field of view level, the present application projects the laser point cloud into a multi-resolution two-dimensional depth map, and filters out falsely detected targets caused by environmental noise in the laser point cloud based on the number of point clouds contained in the depth map within the target area; then, based on the variance of the reflection intensity and distance of each pixel in the depth map within the target area, it is ensured that when a small target partially blocks a large target, the laser point cloud can be associated with the small target and the large target respectively, so as to obtain accurate estimates of the laser radar at different target distances and improve the target positioning accuracy.

[0086] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of this application, "plurality" means two or more, unless otherwise specifically specified.

[0087] In the description of this application, the word "for example" is used to mean "used as an example, illustration or illustration". Any embodiment described in this application as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0088] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A multi-resolution asymmetric fusion target positioning method for laser point cloud and two-dimensional image, characterized in that: The method comprises the following steps: Collect laser point clouds and 2D images; Input the two-dimensional image into a pre-built target detection network model and output the target detection result; wherein the target detection result includes the target area; Perform field-of-view alignment of laser point clouds and 2D images; Project the laser point cloud into a multi-resolution depth map, perform pixel-level alignment on the depth map with the pyramid features of the two-dimensional image, and obtain the array data of each pixel of the depth map after pixel-level alignment; Calculate the distance between the target and the LiDAR based on the array data of each pixel in the depth map within the target area; The pixel array data includes the distance mean, distance variance, average reflection intensity and point cloud quantity; The method for calculating the distance value between the target and the laser radar according to the array data of each pixel of the depth map within the target area includes: calculating the distance value between the target and the laser radar according to the distance variance and average reflection intensity of each pixel of the depth map within the target area; The calculation of the distance between the target and the laser radar includes: Calculate the distance variance and reflection intensity variance of all pixels within the corresponding border of the target in the depth map; If any of the distance variance Vd and reflection intensity variance Vi of all pixels in the target corresponding frame in the depth map is greater than the corresponding specified first threshold and the second threshold This indicates that the corresponding target frame may contain distance information of other targets. So, sort the distances between each pixel of the depth map within the target frame and the LiDAR, divide the minimum and maximum distances into L distance intervals, count the number of pixels falling into each distance interval, and take the middle value of the distance interval with the largest number of pixels as the distance value between the target and the LiDAR. If none of the distance variance Vd and reflection intensity variance Vi of all pixels in the target corresponding frame in the depth map is greater than the corresponding specified first threshold and the second threshold The average distance between all pixels in the depth map within the target frame and the lidar is taken as the distance value between the target and the lidar.

2. The target positioning method of multi-resolution asymmetric fusion of laser point cloud and two-dimensional image according to claim 1 is characterized in that: Before calculating the distance value between the target and the laser radar, it also includes: filtering out falsely detected targets based on the array data of each pixel of the depth map in the target area.

3. The multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image according to claim 2 is characterized in that: The method for filtering misdetected targets according to array data of each pixel of the depth map within the target area includes: filtering misdetected targets according to the number of point clouds in the array data of each pixel of the depth map within the target area.

4. The multi-resolution asymmetric fusion target positioning method of laser point cloud and two-dimensional image according to claim 1 is characterized in that: The method for field-of-view alignment of laser point clouds and two-dimensional images includes the following steps: Filter the laser point cloud outside the camera's horizontal field of view; According to the target detection results, all targets in the target area that are outside the vertical field of view of the lidar are filtered.

5. The target positioning method of multi-resolution asymmetric fusion of laser point cloud and two-dimensional image according to claim 3 is characterized in that: The method for filtering out falsely detected targets based on the number of point clouds in the array data of each pixel of the depth map within the target area includes the following sub-steps: According to the pyramid feature layer corresponding to the target in the target detection result, a depth map of corresponding resolution is selected for each target; wherein different pyramid feature layers correspond to depth maps of different resolutions; According to the target area in the target detection result, count the number of pixels with zero point cloud in the depth map within the target area; Filter out falsely detected targets based on the number of counted pixels.

6. The target positioning method of multi-resolution asymmetric fusion of laser point cloud and two-dimensional image according to claim 5 is characterized in that: If the number of pixels with zero point cloud in the depth map within the target border is greater than the specified threshold, the false detection target is filtered out. Otherwise, there is no need to filter the false detection target.

7. The target positioning method of multi-resolution asymmetric fusion of laser point cloud and two-dimensional image according to any one of claims 1 to 6, characterized in that: Laser point clouds are collected through lidar; two-dimensional images are collected through cameras.

8. The target positioning method of multi-resolution asymmetric fusion of laser point cloud and two-dimensional image according to claim 7 is characterized in that: The method also includes: locating the target from a bird's-eye view according to the distance value between the target and the laser radar and the azimuth angle of the target in the two-dimensional image.

Images