Methods, apparatus and storage media for near-field blind spot perception of autonomous vehicles

By employing techniques such as motion compensation and 3D-to-2D projection conversion, combined with 2D and 3D sensor outputs, the problem of near-field blind spot perception for autonomous vehicles has been solved, achieving end-to-end full-coverage perception and providing an effective reference for the planning and control of autonomous vehicles.

CN118537822BActive Publication Date: 2026-11-17COWA TECHNOLOGY CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410342144.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-25
Publication Date
2026-11-17
Estimated Expiration
2044-03-25

AI Technical Summary

Technical Problem

In existing technologies, near-field blind spot perception of autonomous vehicles has blind spots, making it difficult to detect low-lying targets and increasing the risk of accidents. Existing 2D image detection lacks depth information and cannot be directly used for autonomous driving perception.

Method used

By employing methods such as motion compensation, 3D-to-2D projection conversion, 2D target detection, 3D point extraction, ground point filtering, BEV heatmap generation, target point clustering, IOU calculation, and candidate cluster fusion, combined with 2D and 3D sensor outputs, end-to-end near-field blind zone perception is achieved.

Benefits of technology

It achieves full-coverage perception of near-field blind spots for autonomous vehicles, providing effective reference information for planning and control, and improving vehicle safety and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118537822B_ABST
    Figure CN118537822B_ABST
Patent Text Reader

Abstract

This invention discloses a method, apparatus, and storage medium for near-field blind spot perception of autonomous vehicles, comprising the following steps: motion compensation, 3D to 2D projection conversion, 2D target detection, extraction of target 3D points, filtering of ground points, generation of BEV heatmap, target point clustering, calculation of IOU, selection of optimal candidate clusters, and fusion of candidate clusters. The candidate clusters selected after the previous step for each target are fused to obtain the final target position output. This invention mainly utilizes computer vision technology and post-fusion operations to effectively fuse the outputs of 2D and 3D sensors based on 2D image segmentation, achieving end-to-end near-field blind spot perception full coverage for autonomous vehicles and providing effective reference information for planning and control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a scheme for fusing multimodal information from surround-view fisheye cameras and blind-spot lidar to perceive near-field blind spots of autonomous vehicles, and particularly to a method, apparatus and storage medium for near-field blind spot perception of autonomous vehicles. Background Technology

[0002] In today's era of rapid development of various high technologies, autonomous driving technology has received increasing attention. Various automotive manufacturers are investing heavily in the research and development of Auto Driving Systems (ADS), which encompass a complete workflow from perception to decision-making. The perception system utilizes various sensors installed on the vehicle to perceive the surrounding environment in real time. The data returned by these sensors is then used with advanced algorithms to provide a basis for downstream decision-making tasks.

[0003] The perception system plays a crucial role as the first step in realizing autonomous driving. Currently, most perception systems adopt a multi-sensor fusion solution, including cameras, LiDAR, millimeter-wave radar, ultrasonic radar, etc. Each of these sensors has its own advantages and disadvantages, and the data formats they return vary greatly. Therefore, it is of great significance to leverage the advantages of each sensor and fuse them to output a unified perception result.

[0004] This invention focuses on the near-field blind spots of autonomous vehicles. Currently, in most autonomous vehicle solutions, LiDAR is installed on the roof to prevent its rays from being blocked. However, due to the limitations of the LiDAR's vertical field of view (FOV), there are perception blind spots in the near-field area around the vehicle that the sensor cannot cover. If low-lying objects (such as children, pets, or people who suddenly fall) appear in these blind spots, there is a significant risk of accidents. Therefore, perceiving the near-field blind spots of autonomous vehicles has become an urgent problem to be solved.

[0005] Therefore, existing technologies have developed methods that fuse 2D image detection with radar depth information to achieve near-field blind spot perception for autonomous vehicles. The 2D image detection relies on existing image instance segmentation algorithms, such as the Condinst algorithm, for details of which can be found in the paper "Conditional Convolutions for Instance Segmentation".

[0006] In simple terms, this scheme uses a detection + segmentation approach to identify the mask of each target instance in the image. The specific steps are as follows, where steps 2-4 are the detection branch and steps 5-6 are the segmentation branch:

[0007] 1. Input an RGB three-channel image

[0008] 2. Use a convolutional neural network to downsample the input image and extract features, generating multi-scale feature maps. For example, as shown by the blue quadrilaterals in the first column of the image, C3 is the feature map after downsampling the original image size by 8 times, C4 is 16 times, and C5 is 32 times.

[0009] 3. Use a Feature Pyramid Network (FPN) to fuse and generate multi-scale feature maps, thus aggregating multi-scale information. For example, as shown by the orange quadrilateral in the second column of the figure, the deepest feature map C5 is passed through a convolutional layer to generate a feature map P5 of the same size. Then, downsampling and upsampling are performed respectively. P6 and P7 are feature maps after downsampling P5 by 2 and 4 times, that is, feature maps downsampled by 64 times and 128 times the original image size. P4 is a feature map generated by fusing P5 upsampled by 2 times with the shallow feature C4, and P3 is a feature map generated by fusing P4 upsampled by 2 times with the shallow feature C3. The sizes of P3, P4, and P5 are the same as the sizes of C3, C4, and C5.

[0010] 4. For feature maps of different scales, a parameter-shared detection head is attached to predict the class, location information, and dynamic convolutional kernel for each instance. The convolutional kernels are different for different instances, and are used by the subsequent dynamic convolutional network to predict the mask for each instance. For example, in the shared head in the figure, classification Px,y represents the predicted class, and controller represents the convolutional kernel applicable to each instance.

[0011] 5. Add location-related information to the feature map that incorporates multi-scale information. For example, encode the instance location information predicted in step 4 into a tensor of the same size as P3, and concatenate it with Fmask along the channel dimension.

[0012] 6. Using the convolution kernels of each instance generated in step 4 as the convolution kernels of the convolutional network, input the feature maps from step 5 into the fully convolutional network (FCN) to obtain the pixel-level segmentation results of each instance, i.e., the mask.

[0013] While the aforementioned existing technologies have achieved high accuracy in the industry, they only detect 2D images and lack depth information of the target, therefore they cannot be directly used in the perception modules of autonomous vehicles. We need to fuse the 2D detection results with radar 3D information in some way to obtain end-to-end output. Summary of the Invention

[0014] To address the aforementioned problems of existing technical solutions, this invention proposes a method, apparatus, and storage medium for near-field blind spot perception of autonomous vehicles.

[0015] To achieve the above objectives, the present invention adopts the following technical solution: a method for near-field blind spot perception of autonomous vehicles, comprising the following steps:

[0016] S1, Motion Compensation, is used to compensate for the offset caused by the difference in exposure time between the radar and the camera.

[0017] S2, 3D to 2D projection transformation, projects the motion-compensated radar point cloud from the Baselink coordinate system to the pixel coordinate system on the camera image, obtaining a one-to-one correspondence between each radar point and the image pixel;

[0018] S3, 2D object detection, uses the Condinst model to detect objects of interest in all 2D images and obtain the mask for each object;

[0019] S4, extract the target 3D points. Based on the one-to-one correspondence between each radar point and image pixel obtained in S2, and the 2D mask of each target obtained in S3, extract the points that fall in each target mask as the points of the target in 3D space.

[0020] S5, Filter ground points. Based on the prior size of different target categories, filter the point cloud of the part where the target is connected to the ground to avoid the outer boundary contour of the target being too large in the final 3D space due to residual ground points.

[0021] S6 generates a BEV heatmap (Bird's Eye View, BEV), projecting all 3D point clouds of the target onto the BEV space; multiple 3D points may correspond to the same point in the BEV space;

[0022] S7, Target point clustering: In order to filter out most of the discrete noise points, each point with a heat value in the BEV heat map generated in the previous step is sorted in descending order, and then selected as the starting seed point for clustering in turn, and clustering is performed outward to find the cluster in the BEV grid coordinate system.

[0023] S8, calculate IOU. For the multiple candidate clusters obtained in S7, it is necessary to calculate relevant indicators and select the K clusters with the highest indicators as the global optimal clusters, and filter out noisy clusters.

[0024] S9, Select the optimal candidate cluster. The above operations will calculate C IOU indices for the C clusters clustered in S7, filter out erroneous clusters, and select the optimal cluster. For static targets, the cluster with the top IOU is directly selected. For dynamic targets, the optimal cluster is selected by predicting the target depth value.

[0025] S10, Fusion Candidate Clusters: The candidate clusters selected from the previous step for each target are fused together to obtain the final target position output.

[0026] This invention mainly utilizes computer vision technology and post-fusion operations to effectively fuse the outputs of 2D and 3D sensors based on 2D image segmentation, achieving end-to-end near-field blind spot perception full coverage for autonomous vehicles and providing effective reference information for planning and control.

[0027] Furthermore, step S1 is as follows:

[0028] The UTM (Universal Transverse Mercator) coordinate system is denoted as the world coordinate system, and the vehicle coordinate system is denoted as the Baselink coordinate system. The original radar point cloud is in the Baselink coordinate system and is denoted as Pb. The pose represents the transformation relationship from the Baselink coordinate system to the world coordinate system, which includes the rotation matrix R and the translation vector T. The radar and the camera each have a pose at their respective exposure times, denoted as Rlidar and Tlidar, and Rcamera and Tcamera, respectively. The compensated radar point cloud is denoted as Pb'. Motion compensation is achieved through the following formula:

[0029]

[0030] Where P w =R lidar @P b +T lidar @ indicates matrix multiplication. This indicates finding the inverse of a rotation matrix.

[0031] Furthermore, S2 specifically refers to:

[0032] Using the camera intrinsic parameters K, distortion sparsity D, and camera extrinsic parameters Rc2b and Tc2b, the motion-compensated radar point cloud is projected from the Baselink coordinate system onto the pixel coordinate system of the camera image, obtaining a one-to-one correspondence between each radar point and an image pixel. The projection here uses a common standard projection process; for details, please refer to the `projectPoints` function in the open-source cross-platform computer vision library OpenCV, which will not be elaborated upon here.

[0033] Furthermore, step S6 employs the following method:

[0034] The BEV grid size is set to 5cm x 5cm, meaning the actual distance between two adjacent BEV coordinates is 5cm. To more smoothly represent which locations in the BEV space have the most points, a Gaussian kernel with a radius of 3 is used to calculate the thermal value. That is, the thermal value of each point in the BEV space is related to the value of the points around it that are 3 units away. The denser the points in a region, the more points there are, and the closer they are to the center, the greater the thermal value.

[0035] Furthermore, S7 includes the following method:

[0036] The clustering algorithm chosen is Breadth First Search (BFS), with a search radius of 5cm. The clustering condition is set as follows: the distance between the searched point and the center of the current cluster is less than or equal to half the diagonal distance of the prior size of the current category in the BEV grid (half the length of the red line in the figure below). Points that have already been clustered into a certain cluster will not participate in future clustering. Therefore, theoretically, the earlier the cluster is in the order, the more likely it is to be the target cluster.

[0037] Furthermore, S8 includes the following method:

[0038] Based on the target mask detected by the model in S3, the corresponding target region (ROI) rectangle is extracted and divided into an N x M grid (note that this is image gridding, which is different from BEV gridding in step 7 above). Assuming the rectangle width is W, the height is H, and each grid size is G x G, then N = W / G, M = H / G, both rounded up. Next, the Intersection over Union (IOU) ratio between the 3D radar points and the 2D image at the grid scale is calculated:

[0039] Union (U): After dividing the image ROI into grids, count the number of grids in the image that the 3D radar point cloud falls into after being projected onto the 2D image (refer to step 2);

[0040] Intersection (I): For each cluster in BEV space, count the number of grids in the image that all 3D points in the cluster fall into after being projected onto the 2D image (refer to step 2);

[0041] Finally, IOU is calculated, where IOU = I / U.

[0042] Furthermore, S9 includes the following steps:

[0043] a. The fisheye camera image is distorted based on intrinsic and extrinsic parameters and distortion coefficients to obtain the corrected image;

[0044] b. At this point, the image is distortion-free and can be approximated as a normal camera image, based on the transformation relationship between the camera coordinate system and the pixel coordinate system:

[0045]

[0046] c. If we only consider the X direction, we can obtain: Z c u = f x X c +Z c c x Then depth Where u represents the x-coordinate of the target point in the pixel coordinate system, c x Let uc represent the x-coordinate of the center point of the pixel coordinate system. x This represents the horizontal offset of the target point from the image center in the pixel coordinate system, where fx is the focal length in the horizontal direction, and X is the focal length in the horizontal direction. c If u and x are the x-coordinates of the target point in the camera coordinate system, then u and x can be used to define the x-coordinates. c Replace with the target's width Δu and actual width ΔX in pixel coordinates. c ,Right now Where ΔX c The width of the target of this category can be taken as the approximate prior size in reality, and Δu is the mask width detected by the model in step 3, so that the estimated target depth Z can be obtained. c ;

[0047] d. Alternatively, the incident angle of the target in the X direction in the camera coordinate system can be obtained. Here we only estimate the depth for targets with |α|≤70°, because we are using a fisheye camera, and the distortion at the edge of the image is too severe. After distortion removal, this part of the field of view will be lost.

[0048] e. Since the orientation of the target in the image is unknown, ΔX cannot be determined. c Should we take the width or length from the target's prior dimensions? Here, we assume the target's orientation is the same as or opposite to the vehicle's orientation, and use the following formula to obtain an approximate ΔX. c =P L R+P W (1-R), where P L P represents the prior length of the target. W Indicates the prior width of the target.

[0049] f. The target depth estimated by the above method will deviate from the actual depth. A threshold is set to control the tolerance, and clusters with deviations within the tolerance range are retained. For close targets, we expect the estimated depth to be more accurate, so a lower threshold is set. For distant targets, the depth estimation result will have a larger deviation (a limitation of this method), so a larger threshold is set. Therefore, this threshold can be linearly changed according to the actual position of the target cluster and the distance between it and the vehicle. The farther the distance, the higher the threshold.

[0050] g. Based on IOU and depth prediction, select the optimal candidate cluster for dynamic targets.

[0051] Furthermore, S10 includes the following steps:

[0052] S101. For deformable targets (fences, piles of two-wheeled vehicles, etc.), due to their variable size, multiple candidate clusters need to be considered, and the IOU must be greater than 0.01. Here, the DBSCAN clustering algorithm is used to perform secondary clustering on the clusters. The purpose is to eliminate radar noise clusters between the vehicle and the target, and to cluster up to the top 10 clusters for fusion and output.

[0053] S102: For general targets, if there is only one candidate cluster, output it directly; if there are two or more candidate clusters for static targets or dynamic targets at the edge of the screen whose depth has not been predicted, output the cluster with the highest IOU (Intersection over Union); if there are two or more candidate clusters for dynamic targets and the depth has been estimated, calculate the Euclidean distance between the target center estimated by the depth and the center of the second highest cluster. The smaller the distance, the closer the estimated position is to the actual position. If the distance to the former is less than the distance to the latter, and the distance to the former is <5m, output the former; otherwise, output the latter.

[0054] The present invention also provides an apparatus for near-field blind spot perception of autonomous vehicles, comprising:

[0055] The motion compensation module is used to compensate for the offset caused by the difference in exposure time between the radar and the camera.

[0056] The 3D to 2D projection conversion module projects the motion-compensated radar point cloud from the Baselink coordinate system to the pixel coordinate system on the camera image, obtaining a one-to-one correspondence between each radar point and the image pixel.

[0057] The 2D object detection module uses the Condinst model to detect objects of interest in all 2D images and obtain the mask for each object.

[0058] The target 3D point extraction module extracts the points falling in each target mask as the target's points in 3D space by using the one-to-one correspondence between each radar point and image pixel obtained in S2, and the 2D mask of each target obtained in S3.

[0059] The ground point filtering module filters the point cloud of the part where the target is connected to the ground based on the prior size of different types of targets, so as to avoid the outer boundary contour of the target being too large in the final 3D space due to residual ground points.

[0060] The BEV heatmap module (Bird's Eye View, BEV) projects all 3D point clouds of the target into the BEV space; multiple 3D points may correspond to the same point in the BEV space.

[0061] In order to filter out most of the discrete noise, the target point clustering module sorts each point with a heat value in the BEV heat map generated in the previous step in descending order, and then selects them one by one as the starting seed point for clustering, and searches outward to find the cluster in the BEV grid coordinate system.

[0062] The IOU calculation module needs to calculate relevant indicators for multiple candidate clusters obtained in S7, select the K clusters with the highest indicators as the global optimal clusters, and filter out noisy clusters.

[0063] The optimal candidate cluster selection module calculates C IOU metrics for the C clusters clustered in S7. Due to issues such as occlusion and inaccurate target masks, depth misassociations can occur, leading to incorrect cluster selection. Distant targets may be mistakenly associated with nearby locations (e.g., vegetation, fences; the image above illustrates this, where if the intersection of vegetation in cluster 2 is greater than that in cluster 1, it will be mistakenly associated). In such cases, it's necessary to filter out erroneous clusters and select the optimal cluster. For static targets, since they have no speed and their positions are fixed, misassociating them with nearby locations won't affect the vehicle's decision-making; the top-1 IOU cluster can be directly selected. For dynamic targets, since their speed and position are constantly changing, this will affect the autonomous driving's decision-making logic for a period of time. Therefore, for dynamic targets, we use predicted target depth values ​​to assist in selecting the optimal cluster; (this is a rough depth estimate and won't be very accurate, but it's sufficient for assisting in cluster selection).

[0064] The candidate cluster fusion module merges the candidate clusters selected from the previous step for each target to obtain the final target's position output.

[0065] The present invention also provides a computer-readable storage medium comprising a computer program that, when executed by one or more processors, implements the method for near-field blind spot perception of an autonomous vehicle as described above. Attached Figure Description

[0066] Figure 1 This is a schematic diagram illustrating the generation of the BEV heat map according to the present invention;

[0067] Figure 2 A schematic diagram of clustering for target points;

[0068] Figure 3 This is a diagram illustrating an example of calculating IOU. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] like Figure 1 As shown, a method for near-field blind spot perception in autonomous vehicles includes the following steps:

[0071] S1, Motion Compensation, is used to compensate for the offset caused by the difference in exposure time between the radar and the camera.

[0072] S2, 3D to 2D projection transformation, projects the motion-compensated radar point cloud from the Baselink coordinate system to the pixel coordinate system on the camera image, obtaining a one-to-one correspondence between each radar point and the image pixel;

[0073] S3, 2D object detection, uses the Condinst model to detect objects of interest in all 2D images and obtain the mask for each object;

[0074] S4, extract the target 3D points. Based on the one-to-one correspondence between each radar point and image pixel obtained in S2, and the 2D mask of each target obtained in S3, extract the points that fall in each target mask as the points of the target in 3D space.

[0075] S5, Filter ground points. Based on the prior size of different target categories, filter the point cloud of the part where the target is connected to the ground to avoid the outer boundary contour of the target being too large in the final 3D space due to residual ground points.

[0076] S6 generates a BEV heatmap (Bird's Eye View, BEV), projecting all 3D point clouds of the target onto the BEV space; multiple 3D points may correspond to the same point in the BEV space;

[0077] S7, Target point clustering: In order to filter out most of the discrete noise points, each point with a heat value in the BEV heat map generated in the previous step is sorted in descending order, and then selected as the starting seed point for clustering in turn, and clustering is performed outward to find the cluster in the BEV grid coordinate system.

[0078] S8, calculate IOU. For the multiple candidate clusters obtained in S7, it is necessary to calculate relevant indicators and select the K clusters with the highest indicators as the global optimal clusters, and filter out noisy clusters.

[0079] S9 selects the optimal candidate cluster. The above operations will calculate C IOU metrics for the C clusters clustered in S7. Due to issues such as occlusion and inaccurate target masks, depth misassociation may occur, leading to the selection of incorrect clusters. Distant targets may be mistakenly associated with nearby locations (e.g., vegetation, fences; the above image is an example. If the intersection of vegetation in cluster 2 is greater than that in cluster 1, it will be mistakenly associated). In this case, it is necessary to filter out erroneous clusters and select the optimal cluster. For static targets, since they have no speed and their positions are fixed, misassociating them with nearby locations will not affect the vehicle's decision-making; the top 1 IOU cluster can be directly selected. For dynamic targets, since their speed exists and their positions are constantly changing, this will affect the autonomous driving's decision-making logic for a period of time. Therefore, for dynamic targets, we use the prediction of target depth values ​​to assist in selecting the optimal cluster; (this is a rough estimate of the depth and will not be very accurate, but it is sufficient to assist in cluster selection).

[0080] S10, Fusion Candidate Clusters: The candidate clusters selected from the previous step for each target are fused together to obtain the final target position output.

[0081] This invention mainly utilizes computer vision technology and post-fusion operations to effectively fuse the outputs of 2D and 3D sensors based on 2D image segmentation, achieving end-to-end near-field blind spot perception full coverage for autonomous vehicles and providing effective reference information for planning and control.

[0082] In practical applications, due to the difference in exposure time between radar and camera, if the vehicle is moving during this time difference, direct projection will cause the result to be offset. Therefore, motion compensation is required for the projection process based on the vehicle's pose. The pose is obtained through GPS and IMU sensors, which will not be elaborated here.

[0083] The steps of motion compensation are as follows:

[0084] The UTM (Universal Transverse Mercator) coordinate system is denoted as the world coordinate system, the vehicle coordinate system is denoted as the Baselink coordinate system, and the original radar point cloud is denoted as P in the Baselink coordinate system. b The pose represents the transformation from the Baselink coordinate system to the world coordinate system, including the rotation matrix R and the translation vector T. The radar and camera each have a pose at their respective exposure times, denoted as R. lidar T lidar and R camera T camera The compensated radar point cloud is denoted as P. b Motion compensation is achieved through the following formula:

[0085]

[0086] Where P w =R lidar @P b +T lidar @ indicates matrix multiplication. This indicates finding the inverse of a rotation matrix.

[0087] Specifically, S2 is:

[0088] Using camera intrinsic parameters K, distortion sparsity D, and camera extrinsic parameters R c2b T c2b The motion-compensated radar point cloud is projected from the Baselink coordinate system onto the pixel coordinate system of the camera image, obtaining a one-to-one correspondence between each radar point and an image pixel. The projection here uses a common standard projection process; for details, please refer to the `projectPoints` function in the open-source cross-platform computer vision library OpenCV, which will not be elaborated upon here.

[0089] The S6 example is as follows:

[0090] The BEV grid size is set to 5cm x 5cm, meaning the actual distance between two adjacent BEV coordinates is 5cm. To more smoothly represent which locations in the BEV space have the most points, a Gaussian kernel with a radius of 3 is used to calculate the thermal values. This means that the thermal value of each point in the BEV space is related to the values ​​of points within a radius of 3 around it; the denser the points in a region, the more points there are, and the closer they are to the center, the greater the thermal value. Figure 1 As shown.

[0091] To filter out most discrete noise points, the points with thermal values ​​in the BEV heatmap generated in the previous step are sorted in descending order. Then, each point is selected sequentially as a starting seed point for clustering, and clustering is performed outwards to find clusters in the BEV grid coordinate system. The target point clustering includes the following methods:

[0092] The clustering algorithm chosen is Breadth-First Search (BFS), with a search radius of 5 cm. The clustering condition is set as follows: the distance between the searched point and the current cluster center is ≤ half the diagonal distance of the prior size of the current category in the BEV grid (e.g., ...). Figure 2 Points that have already been clustered into a certain cluster (as shown by half the length of the red line) will not participate in clustering again in the future. Therefore, theoretically, the earlier the cluster appears in the sequence, the more likely it is to be the target cluster.

[0093] The above clustering steps will yield multiple candidate clusters. Relevant metrics need to be calculated for each cluster, and the K clusters with the highest metrics are selected as the globally optimal clusters, filtering out noisy clusters. The IOU calculation includes the following methods:

[0094] Based on the target mask detected by the model in S3, the corresponding target region (ROI) rectangle is extracted and divided into an N x M grid (note that this is image gridding, which is different from BEV gridding in step 7 above). Assuming the rectangle width is W, the height is H, and each grid size is G x G, then N = W / G, M = H / G, both rounded up. Next, the Intersection over Union (IOU) ratio between the 3D radar points and the 2D image at the grid scale is calculated:

[0095] Union (U): After dividing the image ROI into grids, count the number of grids in the image that the 3D radar point cloud falls into after being projected onto the 2D image (refer to step 2);

[0096] Intersection (I): For each cluster in BEV space, count the number of grids in the image that all 3D points in the cluster fall into after being projected onto the 2D image (refer to step 2); see Figure 3 As shown.

[0097] Finally, IOU is calculated, where IOU = I / U.

[0098] The above operations will calculate C IOU metrics for the C clusters clustered in S7. Due to issues such as occlusion and inaccurate target masks, depth misassociations may occur, leading to the selection of incorrect clusters and misassociating distant targets with nearby locations (such as vegetation, fences, etc.). Figure 3For example, if the intersection of vegetation in cluster 2 is greater than that in cluster 1, it will be mistakenly associated. In this case, it is necessary to filter out the erroneous clusters and select the optimal cluster. For static targets, since they have no speed and their positions are fixed, mistaken association with nearby targets will not affect the autonomous vehicle's decision-making, and the top 1 IOU cluster can be directly selected. However, for dynamic targets, since they have speed and their positions are constantly changing, this will affect the decision-making logic of the autonomous vehicle in the future. Therefore, for dynamic targets, we use the prediction of the target depth value to assist in selecting the optimal cluster (here, it is an approximate depth estimate, which will not be very accurate, but it is sufficient to assist in cluster selection). S9 includes the following steps:

[0099] a. The fisheye camera image is distorted based on intrinsic and extrinsic parameters and distortion coefficients to obtain the corrected image;

[0100] b. At this point, the image is distortion-free and can be approximated as a normal camera image, based on the transformation relationship between the camera coordinate system and the pixel coordinate system:

[0101]

[0102] c. If we only consider the X direction, we can obtain: Z c u = f x X c +Z c c x Then depth Where u represents the x-coordinate of the target point in the pixel coordinate system, c x Let uc represent the x-coordinate of the center point of the pixel coordinate system. x This represents the horizontal offset of the target point from the image center in the pixel coordinate system, where fx is the focal length in the horizontal direction, and X is the focal length in the horizontal direction. c If u and x are the x-coordinates of the target point in the camera coordinate system, then u and x can be used to define the x-coordinates. c Replace with the target's width Δu and actual width ΔX in pixel coordinates. c ,Right now Where ΔX c The width of the target of this category can be taken as the approximate prior size in reality, and Δu is the mask width detected by the model in step 3, so that the estimated target depth Z can be obtained. c ;

[0103] d. Alternatively, the incident angle of the target in the X direction in the camera coordinate system can be obtained. Here we only estimate the depth for targets with |α|≤70°, because we are using a fisheye camera, and the distortion at the edge of the image is too severe. After distortion removal, this part of the field of view will be lost.

[0104] e. Since the orientation of the target in the image is unknown, ΔX cannot be determined. c Should we take the width or length from the target's prior dimensions? Here, we assume the target's orientation is the same as or opposite to the vehicle's orientation, and use the following formula to obtain an approximate ΔX. c =P L R+P W (1-R), where P L P represents the prior length of the target. W Indicates the prior width of the target.

[0105] f. The target depth estimated by the above method will deviate from the actual depth. A threshold is set to control the tolerance, and clusters with deviations within the tolerance range are retained. For close targets, we expect the estimated depth to be more accurate, so a lower threshold is set. For distant targets, the depth estimation result will have a larger deviation (a limitation of this method), so a larger threshold is set. Therefore, this threshold can be linearly changed according to the actual position of the target cluster and the distance between it and the vehicle. The farther the distance, the higher the threshold.

[0106] g. Based on IOU and depth prediction, select the optimal candidate cluster for dynamic targets.

[0107] Furthermore, S10 includes the following steps:

[0108] S101. For deformable targets (fences, piles of two-wheeled vehicles, etc.), due to their variable size, multiple candidate clusters need to be considered, and the IOU must be greater than 0.01. Here, the DBSCAN clustering algorithm is used to perform secondary clustering on the clusters. The purpose is to eliminate radar noise clusters between the vehicle and the target, and to cluster up to the top 10 clusters for fusion and output.

[0109] S102: For general targets, if there is only one candidate cluster, output it directly; if there are two or more candidate clusters for static targets or dynamic targets at the edge of the screen whose depth has not been predicted, output the cluster with the highest IOU (Intersection over Union); if there are two or more candidate clusters for dynamic targets and the depth has been estimated, calculate the Euclidean distance between the target center estimated by the depth and the center of the second highest cluster. The smaller the distance, the closer the estimated position is to the actual position. If the distance to the former is less than the distance to the latter, and the distance to the former is <5m, output the former; otherwise, output the latter.

[0110] This patent mainly utilizes computer vision technology and post-fusion operations to effectively integrate the outputs of 2D and 3D sensors based on 2D image segmentation, achieving end-to-end near-field blind spot perception full coverage for autonomous vehicles and providing effective reference information for planning and control.

[0111] The present invention also provides an apparatus for near-field blind spot perception of autonomous vehicles, comprising:

[0112] The motion compensation module is used to compensate for the offset caused by the difference in exposure time between the radar and the camera.

[0113] The 3D to 2D projection conversion module projects the motion-compensated radar point cloud from the Baselink coordinate system to the pixel coordinate system on the camera image, obtaining a one-to-one correspondence between each radar point and the image pixel.

[0114] The 2D object detection module uses the Condinst model to detect objects of interest in all 2D images and obtain the mask for each object.

[0115] The target 3D point extraction module extracts the points falling in each target mask as the target's points in 3D space by using the one-to-one correspondence between each radar point and image pixel obtained in S2, and the 2D mask of each target obtained in S3.

[0116] The ground point filtering module filters the point cloud of the part where the target is connected to the ground based on the prior size of different types of targets, so as to avoid the outer boundary contour of the target being too large in the final 3D space due to residual ground points.

[0117] The BEV heatmap module (Bird's Eye View, BEV) projects all 3D point clouds of the target into the BEV space; multiple 3D points may correspond to the same point in the BEV space.

[0118] In order to filter out most of the discrete noise, the target point clustering module sorts each point with a heat value in the BEV heat map generated in the previous step in descending order, and then selects them one by one as the starting seed point for clustering, and searches outward to find the cluster in the BEV grid coordinate system.

[0119] The IOU calculation module needs to calculate relevant indicators for multiple candidate clusters obtained in S7, select the K clusters with the highest indicators as the global optimal clusters, and filter out noisy clusters.

[0120] The optimal candidate cluster selection module calculates C IOU metrics for the C clusters clustered in S7. Due to issues such as occlusion and inaccurate target masks, depth misassociations can occur, leading to incorrect cluster selection. Distant targets may be mistakenly associated with nearby locations (e.g., vegetation, fences; the image above illustrates this, where if the intersection of vegetation in cluster 2 is greater than that in cluster 1, it will be mistakenly associated). In such cases, it's necessary to filter out erroneous clusters and select the optimal cluster. For static targets, since they have no speed and their positions are fixed, misassociating them with nearby locations won't affect the vehicle's decision-making; the top-1 IOU cluster can be directly selected. For dynamic targets, since their speed and position are constantly changing, this will affect the autonomous driving's decision-making logic for a period of time. Therefore, for dynamic targets, we use predicted target depth values ​​to assist in selecting the optimal cluster; (this is a rough depth estimate and won't be very accurate, but it's sufficient for assisting in cluster selection).

[0121] The candidate cluster fusion module merges the candidate clusters selected from the previous step for each target to obtain the final target's position output.

[0122] The present invention also provides a computer-readable storage medium comprising a computer program that, when executed by one or more processors, implements the method for near-field blind spot perception of an autonomous vehicle as described above.

[0123] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for automatic driving vehicle near field blind zone perception, characterized in that, Includes the following steps: S1, Motion Compensation, is used to compensate for the offset of moving targets caused by the difference in exposure time between radar and camera; S2, 3D to 2D projection transformation, projects the motion-compensated radar point cloud from the Baselink coordinate system to the pixel coordinate system on the camera image, obtaining a one-to-one correspondence between each radar point and the image pixel; S3, 2D object detection, uses the Condinst model to detect objects of interest in all 2D images and obtain the mask for each object; S4, extract the target 3D points. Based on the one-to-one correspondence between each radar point and image pixel obtained in S2, and the 2D mask of each target obtained in S3, extract the points that fall in each target mask as the points of the target in 3D space. S5, Filter ground points. Based on the prior size of different target categories, filter the point cloud of the part where the target is connected to the ground to avoid the outer boundary contour of the target being too large in the final 3D space due to residual ground points. S6 generates a BEV heatmap, projecting all 3D point clouds of the target onto the BEV space; multiple 3D points may correspond to the same point in the BEV space. S7, Target point clustering: In order to filter out most of the discrete noise points, each point with a heat value in the BEV heat map generated in the previous step is sorted in descending order, and then selected as the starting seed point for clustering in turn, and clustering is performed outward to find clusters in the BEV grid coordinate system. S8, calculate IOU. For the multiple candidate clusters obtained in S7, it is necessary to calculate relevant indicators and select the K clusters with the highest indicators as the global optimal clusters, and filter out noisy clusters. S9, Select the best candidate cluster. The above operations will calculate C IOU indices for the C clusters clustered in S7, filter out erroneous clusters, and select the best cluster; for static targets, directly select the cluster with the top IOU; For dynamic targets, the optimal cluster can be selected by predicting the target depth value; S10, Merge candidate clusters: Merge the candidate clusters selected from the previous step for each target to obtain the final target position output; S9 includes the following steps: a. The fisheye camera image is distorted based on intrinsic and extrinsic parameters and distortion coefficients to obtain the corrected image; b. At this point, the image is distortion-free and can be approximated as a normal camera image, based on the transformation relationship between the camera coordinate system and the pixel coordinate system: ; c. If we only consider the X direction, we get: Then depth ,in This represents the x-coordinate of the target point in the pixel coordinate system. Let x represent the x-coordinate of the center point of the pixel coordinate system. This represents the horizontal offset of the target point from the image center point in the pixel coordinate system. It is the focal length in the horizontal direction. If it is the x-coordinate of the target point in the camera coordinate system, then... and Replace with the width of the target in pixel coordinates and actual width ,Right now ,in Take the width of the approximate prior size of the target in reality for that category. The estimated target depth can be obtained by using the mask width detected by the model in S3. ; d. Determine the incident angle of the target in the X direction in the camera coordinate system. And only for The target estimated depth; e. Assuming the target orientation is the same as or opposite to the vehicle's orientation, use the following formula to obtain an approximate value. ,in Indicates the prior length of the target. Indicates the prior width of the target. ; f. Since there is a deviation between the estimated target depth and the actual depth, a threshold for controlling tolerance is set to retain clusters whose deviation is within the tolerance range; for close targets, a smaller threshold is set; for distant targets, a larger threshold is set. This threshold changes linearly according to the actual position of the target cluster and the distance between it and the vehicle. The farther the distance, the higher the threshold. g. Based on IOU and depth prediction, select the optimal candidate cluster for dynamic targets.

2. The method for near-field blind spot perception of autonomous vehicles according to claim 1, characterized in that, The steps in S1 are as follows: The UTM coordinate system is denoted as the world coordinate system, the vehicle coordinate system as the Baselink coordinate system, and the original radar point cloud is denoted as the Baselink coordinate system. The pose represents the transformation from the Baselink coordinate system to the world coordinate system, including the rotation matrix R and the translation vector T. The radar and camera each have a pose at their respective exposure times, denoted as R. lidar T lidar and R camera T camera The compensated radar point cloud is denoted as Motion compensation is achieved through the following formula: ; in , This represents the matrix multiplication operation. This indicates finding the inverse of a rotation matrix.

3. The method for near-field blind spot perception of autonomous vehicles according to claim 1, characterized in that, Specifically, S2 is: Using camera intrinsic parameters K, distortion coefficients D, and camera extrinsic parameters The motion-compensated radar point cloud is projected from the Baselink coordinate system onto the pixel coordinate system of the camera image, obtaining a one-to-one correspondence between each radar point and an image pixel. The projection here uses a common standard projection process, which can be found in the projectPoints projection function of the open-source cross-platform computer vision library OpenCV.

4. The method for near-field blind spot perception of autonomous vehicles according to claim 1, characterized in that, The S6 method is as follows: The BEV grid size is set to 5cm x 5cm, meaning the actual distance between two adjacent BEV coordinates is 5cm. A Gaussian kernel with a radius of 3 is used to calculate the thermal value. This means that the thermal value of each point in the BEV space is related to the value of the points around it that are 3 units away. The denser the points in a region, the more points there are, and the closer they are to the center, the greater the thermal value.

5. The method for near-field blind spot perception of autonomous vehicles according to claim 1, characterized in that, S7 includes the following methods: The clustering algorithm chosen is Breadth-First Search (BFS), with a search radius of 5cm. The clustering condition is set as follows: the distance between the searched point and the current cluster center is less than or equal to half the diagonal distance of the prior size of the current category in the BEV grid. Points that have already been clustered into a certain cluster will not participate in future clustering. Therefore, the earlier the cluster is in the order, the more likely it is to be the target cluster.

6. The method for near-field blind spot perception of autonomous vehicles according to claim 1, characterized in that, S8 includes the following methods: Based on the target mask detected by the model in S3, the corresponding target region ROI rectangle is extracted and divided into an N x M grid. Assuming the width of the rectangle is W, the height is H, and each grid size is G x G, then N = W / G, M = H / G, both rounded up. Next, the Intersection over Union (IOU) index of the 3D radar points and the 2D image is calculated at the grid scale: Union U: After dividing the image ROI into grids, count the number of grids that the 3D radar point cloud falls into when projected onto the 2D image; Intersection I: For each cluster in BEV space, count the number of grids in the image that all 3D points in the cluster fall into after being projected onto the 2D image; Finally, IOU is calculated: IOU = I / U.

7. The method for near-field blind spot perception of autonomous vehicles according to claim 1, characterized in that, S10 includes the following steps: S101. For deformable targets, due to their variable size, multiple candidate clusters need to be considered, and the IOU must be greater than 0.

01. Here, the DBSCAN clustering algorithm is used to perform secondary clustering on the clusters. The purpose is to eliminate radar noise clusters between the vehicle and the target, and to cluster up to the top 10 clusters for fusion and output. S102: For general targets, if there is only one candidate cluster, output it directly; if there are two or more candidate clusters for static targets or dynamic targets at the edge of the screen whose depth has not been predicted, output the cluster with the highest IOU (Intersection over Union); if there are two or more candidate clusters for dynamic targets and the depth has been estimated, calculate the Euclidean distance between the target center estimated by the depth and the center of the second highest cluster. The smaller the distance, the closer the estimated position is to the actual position. If the distance to the former is less than the distance to the latter, and the distance to the former is < 5m, output the former; otherwise, output the latter.

8. A device for near-field blind spot perception of autonomous vehicles, characterized in that, include: The motion compensation module is used to compensate for the offset of moving targets caused by the difference in exposure time between the radar and the camera; The 3D to 2D projection conversion module projects the motion-compensated radar point cloud from the Baselink coordinate system to the pixel coordinate system on the camera image, obtaining a one-to-one correspondence between each radar point and the image pixel. The 2D object detection module uses the Condinst model to detect objects of interest in all 2D images and obtain the mask for each object. The target 3D point extraction module uses the one-to-one correspondence between each radar point and image pixel obtained from the 3D to 2D projection conversion module, and the 2D mask of each target obtained from the 2D target detection module, to extract the points falling in each target mask as the points of the target in 3D space. The ground point filtering module filters the point cloud of the part where the target is connected to the ground based on the prior size of different types of targets, so as to avoid the outer boundary contour of the target being too large in the final 3D space due to residual ground points. The BEV heatmap generation module projects all 3D point clouds of the target onto the BEV space; multiple 3D points may correspond to the same point in the BEV space. In order to filter out most of the discrete noise, the target point clustering module sorts each point with a thermal value in the BEV heatmap generated in the previous step in descending order, and then selects them one by one as the starting seed point for clustering, and searches outward to find clusters in the BEV grid coordinate system. The IOU calculation module needs to calculate relevant indicators for multiple candidate clusters obtained from the target point clustering module, select the K clusters with the highest indicators as the global optimal clusters, and filter out noisy clusters. The optimal candidate cluster selection module calculates C IOU indices for the C clusters clustered in the target point clustering module, filters out erroneous clusters, and selects the optimal cluster. For static targets, the cluster with the top IOU is directly selected. For dynamic targets, the optimal cluster is selected by predicting the target depth value. The candidate cluster fusion module merges the candidate clusters selected by the optimal candidate cluster selection module for each target to obtain the final target position output; The module for selecting the optimal candidate cluster is used to perform the following steps: a. The fisheye camera image is distorted based on intrinsic and extrinsic parameters and distortion coefficients to obtain the corrected image; b. At this point, the image is distortion-free and can be approximated as a normal camera image, based on the transformation relationship between the camera coordinate system and the pixel coordinate system: ; c. If we only consider the X direction, we get: Then depth ,in This represents the x-coordinate of the target point in the pixel coordinate system. Let x represent the x-coordinate of the center point of the pixel coordinate system. This represents the horizontal offset of the target point from the image center point in the pixel coordinate system. It is the focal length in the horizontal direction. If it is the x-coordinate of the target point in the camera coordinate system, then... and Replace with the width of the target in pixel coordinates and actual width ,Right now ,in Take the width of the approximate prior size of the target in reality for that category. The estimated target depth can be obtained by using the mask width detected by the model in S3. ; d. Determine the incident angle of the target in the X direction in the camera coordinate system. And only for The target estimated depth; e. Assuming the target orientation is the same as or opposite to the vehicle's orientation, use the following formula to obtain an approximate value. ,in Indicates the prior length of the target. Indicates the prior width of the target. ; f. Since there is a deviation between the estimated target depth and the actual depth, a threshold for controlling tolerance is set to retain clusters whose deviation is within the tolerance range; for close targets, a smaller threshold is set; for distant targets, a larger threshold is set. This threshold changes linearly according to the actual position of the target cluster and the distance between it and the vehicle. The farther the distance, the higher the threshold. g. Based on IOU and depth prediction, select the optimal candidate cluster for dynamic targets.

9. A computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by one or more processors, it implements the method for near-field blind spot perception of an autonomous vehicle as described in any one of claims 1-7.

Citation Information

Patent Citations

  • 3D target detection and tracking method based on camera and laser radar

    CN112487919A

  • Automatic driving multi-mode cooperative sensing method and system based on BEV visual angle

    CN116977963A