Occlusion analysis method, device, equipment and storage medium for integrating point cloud and image

Abstract

The present invention relates to the field of computer vision technology, and discloses an occlusion analysis method, device, equipment and storage medium for fusing point cloud and image, which are used to improve the accuracy of occlusion judgment. The occlusion analysis method for fusing point cloud and image includes: obtaining initial point cloud data and camera image, as well as internal and external parameter calibration results of radar and camera; converting and processing the initial point cloud data according to the calibration results to obtain target point cloud data under the camera's perspective; performing edge detection and feature extraction processing on the camera image to obtain image edge information and feature information; performing edge consistency analysis and feature matching processing based on the image edge information, feature information and target point cloud data, identifying and eliminating target point cloud data in the occluded area, and obtaining optimized target point cloud data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an occlusion analysis method, device, equipment and storage medium for fusing point cloud and image. Background Art

[0002] There are many existing occlusion analysis methods: the distance method relies on accurate depth information and is prone to misjudgment in complex scenes; the field of view matching method reduces the accuracy of occlusion judgment and consumes large computing resources in dynamic scenes or complex backgrounds, and when there is a large difference in field of view; the object detection method has high computing resources and is easily affected by lighting and angles. These methods often cannot accurately solve the problem of point cloud mismatching in occluded areas, especially in complex environments, where the accuracy of occlusion judgment is low. Summary of the invention

[0003] The present invention provides an occlusion analysis method, device, equipment and storage medium for fusing point cloud and image, so as to solve the problem of low accuracy of occlusion judgment in the prior art.

[0004] A first aspect of the present invention provides an occlusion analysis method for fusing point clouds and images, comprising: obtaining initial point cloud data and a camera image, as well as calibration results of internal and external parameters of a radar and a camera, wherein the initial point cloud data is point cloud data of a target environment collected by a radar, and the camera image is image data of the target environment collected by a camera; converting the initial point cloud data according to the calibration results to obtain target point cloud data from a camera perspective; performing edge detection and feature extraction on the camera image to obtain image edge information and feature information; performing edge consistency analysis and feature matching based on the image edge information, the feature information, and the target point cloud data to identify and eliminate target point cloud data in an occluded area to obtain optimized target point cloud data.

[0005] In a feasible implementation, the initial point cloud data is converted according to the calibration result to obtain target point cloud data under the camera perspective, including: using the internal and external parameters in the calibration result to calculate the transformation matrix from the radar coordinate system to the camera coordinate system; applying the transformation matrix to convert the initial point cloud data into target point cloud data under the camera coordinate system.

[0006] In a feasible implementation manner, the edge detection and feature extraction processing of the camera image to obtain image edge information and feature information includes: performing image segmentation on the camera image to divide the camera image into multiple regions; applying a corresponding edge detection algorithm and feature extraction algorithm to each region to extract edge information and feature points; integrating the edge information and feature points of each region to obtain image edge information and feature information corresponding to the camera image.

[0007] In a feasible implementation, the application of corresponding edge detection algorithms and feature extraction algorithms to each region to extract image edge information and feature points includes: selecting and applying corresponding edge detection algorithms to extract edge information of each region based on the content, structure or application scenario characteristics of each region; selecting and applying corresponding feature extraction algorithms to extract feature points of each region based on the image characteristics contained in each region or the targets to be identified.

[0008] In a feasible implementation manner, edge consistency analysis and feature matching processing are performed based on the image edge information, the feature information, and the target point cloud data to identify and eliminate the target point cloud data in the occluded area to obtain optimized target point cloud data, including: aligning the image edge information with the target point cloud data; performing edge consistency analysis to compare the geometric consistency of the image edge information with the corresponding edge in the target point cloud data to identify areas with discontinuous or misaligned edges due to occlusion; matching the feature information in the camera image with the feature information in the target point cloud data; identifying and marking the target point cloud data in the occluded area based on the feature matching results and the edge consistency analysis results; and eliminating the target point cloud data marked as the occluded area to obtain optimized target point cloud data.

[0009] In a feasible implementation, the edge consistency analysis is performed to compare the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identify areas with edge discontinuity or misalignment caused by occlusion, including: performing edge extraction on the target point cloud data to form point cloud edge information; aligning the image edge information with the point cloud edge information; using a geometric consistency evaluation algorithm to calculate the deviation between the image edge and the point cloud edge based on the image edge information and the point cloud edge information; and identifying areas with edge discontinuity or misalignment caused by occlusion based on the geometric consistency evaluation result.

[0010] In a feasible implementation, the target point cloud data marked as occluded areas are eliminated to obtain optimized target point cloud data, including: classifying the occluded areas according to the shape, size and position information of the occluded areas; matching different culling strategies for different types of occluded areas; and performing culling processing on the target point cloud data of each occluded area based on each culling strategy to obtain optimized target point cloud data.

[0011] The second aspect of the present invention provides an occlusion analysis device for fusing point clouds and images, comprising: an acquisition module, used to acquire initial point cloud data and camera images, and calibration results of internal and external parameters of radar and camera, wherein the initial point cloud data is point cloud data of a target environment collected by a radar, and the camera image is image data of a target environment collected by a camera; a processing module, used to convert and process the initial point cloud data according to the calibration results to obtain target point cloud data from a camera perspective; an extraction module, used to perform edge detection and feature extraction processing on the camera image to obtain image edge information and feature information; and a elimination module, used to perform edge consistency analysis and feature matching processing based on the image edge information, the feature information and the target point cloud data, identify and eliminate target point cloud data in an occluded area, and obtain optimized target point cloud data.

[0012] In a feasible implementation, the processing module is specifically used to: calculate the transformation matrix from the radar coordinate system to the camera coordinate system using the internal and external parameters in the calibration result; and apply the transformation matrix to convert the initial point cloud data into target point cloud data in the camera coordinate system.

[0013] In a feasible implementation, the extraction module includes: a segmentation unit, used to perform image segmentation on the camera image, dividing the camera image into multiple regions; an extraction unit, used to apply corresponding edge detection algorithms and feature extraction algorithms to each region to extract edge information and feature points; an integration unit, used to integrate the edge information and feature points of each region to obtain image edge information and feature information corresponding to the camera image.

[0014] In a feasible implementation, the extraction unit is specifically used to: select and apply a corresponding edge detection algorithm to extract edge information of each region according to the content, structure or application scenario characteristics of each region; select and apply a corresponding feature extraction algorithm to extract feature points of each region according to the image characteristics contained in each region or the target to be identified.

[0015] In a feasible implementation, the rejection module includes: an alignment unit, used to align the image edge information with the target point cloud data; an analysis unit, used to perform edge consistency analysis, compare the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identify areas with discontinuous or misaligned edges due to occlusion; a matching unit, used to match the feature information in the camera image with the feature information in the target point cloud data; a marking unit, used to identify and mark the target point cloud data in the occluded area based on the feature matching results and the edge consistency analysis results; and a rejection unit, used to reject the target point cloud data marked as the occluded area to obtain optimized target point cloud data.

[0016] In a feasible implementation, the analysis unit is specifically used to: perform edge extraction on the target point cloud data to form point cloud edge information; align the image edge information with the point cloud edge information; use a geometric consistency evaluation algorithm to calculate the deviation between the image edge and the point cloud edge based on the image edge information and the point cloud edge information; and identify areas with discontinuous or misaligned edges due to occlusion based on the geometric consistency evaluation results.

[0017] In a feasible implementation, the culling unit is specifically used to: classify the occluded areas according to the shape, size and position information of the occluded areas; match different culling strategies to different types of occluded areas; and perform culling processing on the target point cloud data of each occluded area based on each culling strategy to obtain optimized target point cloud data.

[0018] The third aspect of the present invention provides an occlusion analysis device for fusing point clouds and images, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the occlusion analysis device for fusing point clouds and images executes the above-mentioned occlusion analysis method for fusing point clouds and images.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned occlusion analysis method for fusing point clouds and images.

[0020] In the technical solution provided by the present invention, initial point cloud data and camera images, as well as the calibration results of internal and external parameters of radar and camera are obtained, the initial point cloud data is the point cloud data of the target environment collected by the radar, and the camera image is the image data of the target environment collected by the camera; the initial point cloud data is converted and processed according to the calibration results to obtain the target point cloud data under the camera's perspective; the camera image is subjected to edge detection and feature extraction processing to obtain image edge information and feature information; edge consistency analysis and feature matching processing are performed according to the image edge information, the feature information and the target point cloud data, the target point cloud data in the occluded area is identified and eliminated, and the optimized target point cloud data is obtained. In the embodiment of the present invention, by using the initial point cloud data collected by the radar, the image data collected by the camera and the calibration results of the internal and external parameters of the radar and camera, the initial point cloud data is converted to the camera's perspective, and edge consistency analysis and feature matching are performed with the camera image, so that the inconsistent area between the image and the point cloud can be more accurately identified, thereby effectively eliminating the target point cloud data in the occluded area, improving the accuracy of occlusion judgment, and obtaining more optimized and accurate target point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an embodiment of an occlusion analysis method for fusing point cloud and image in an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of another embodiment of the occlusion analysis method for fusing point cloud and image in an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of another embodiment of the occlusion analysis method for fusing point cloud and image in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of an embodiment of an occlusion analysis device for fusing point cloud and image in an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of another embodiment of the occlusion analysis device for fusing point cloud and image in an embodiment of the present invention;

[0026] Figure 6 Schematic diagram of an embodiment of an occlusion analysis device for fusing point cloud and image in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present invention provide an occlusion analysis method, device, equipment and storage medium for fusing point cloud and image. Through the fusion analysis of point cloud and image, the accuracy of occlusion judgment is improved, thereby improving the accuracy of the final point cloud data.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It is understandable that the execution subject of the present invention may be an occlusion analysis device for fusing point clouds and images, or a terminal or a server, which is not specifically limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0030] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1, an embodiment of the occlusion analysis method for fusing point cloud and image in the embodiment of the present invention includes:

[0031] 101. Acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera;

[0032] The initial point cloud data can be collected by the LiDAR sensor. The LiDAR sensor measures the distance, angle and other information of objects in the environment by emitting laser pulses and receiving the reflected signals, and then constructs a set of points in three-dimensional space. These point cloud data not only have high-precision geometric information, but also can finely depict the shape, structure and details of the target environment. At the same time, the camera captures the reflected light in the environment to generate a two-dimensional camera image. The camera image not only contains rich color information, but also shows the texture of the object, lighting changes and more complex visual features. In order to ensure that the two types of data can accurately correspond and merge, the internal and external parameters between the radar and the camera must be calibrated. The calibration process includes determining the relative position relationship between the camera and the radar, that is, the external parameters; at the same time, it is also necessary to obtain their respective internal parameters, such as the focal length, principal point coordinates, distortion coefficient, etc. of the camera, as well as the scanning angle and resolution of the radar.

[0033] 102. Convert the initial point cloud data according to the calibration result to obtain target point cloud data under the camera perspective;

[0034] During the data conversion process, the initial point cloud data collected by the radar is converted from the radar coordinate system to the camera coordinate system through the internal and external parameters in the calibration results. This conversion process usually involves the transformation of the three-dimensional coordinate system, including operations such as translation and rotation. The translation operation moves the point cloud data in space to the position of the camera based on the relative position relationship between the radar and the camera, and the rotation operation rotates the point cloud data in space to the camera's perspective based on the relative direction relationship between the radar and the camera.

[0035] During the conversion process, the camera distortion problem also needs to be considered. Since the camera lens has distortion, such as barrel distortion and pincushion distortion, this will cause the pixel position in the image to deviate from the position of the actual object in three-dimensional space. Therefore, when converting point cloud data, the camera distortion parameters need to be used to correct the image distortion to ensure the accurate correspondence between the point cloud data and the image data.

[0036] In addition, in order to obtain more accurate target point cloud data, some optimization algorithms can also be used. For example, the least squares method and other optimization algorithms can be used to smooth the converted point cloud data to reduce the impact of noise and errors. At the same time, filtering algorithms and other methods can be used to further optimize and process the point cloud data to improve the accuracy and reliability of the data.

[0037] 103. Perform edge detection and feature extraction processing on the camera image to obtain image edge information and feature information;

[0038] The edge detection process relies on edge detection algorithms, such as the Canny edge detection algorithm, which accurately depicts the contour of an object by calculating the local maximum of the image gradient, or the Sobel operator, which uses the gradient information in the horizontal and vertical directions to emphasize the edge features in the image, thereby obtaining the edge information of the image. The feature extraction step focuses on mining unique and highly recognizable feature points or feature areas from the image. This process usually relies on algorithms such as the Harris corner detection algorithm, which can identify corner features in the image. These corners are often the most drastically changing points in the image; or the SIFT algorithm, which extracts local features in the image by constructing a scale space and searching for extreme points at different scales. These features are robust to rotation, scale changes, and even partial occlusion; the SURF algorithm accelerates the feature extraction process through integral images and Haar wavelets, while maintaining a high feature matching accuracy. During the feature extraction process, each feature point or region is assigned a unique feature descriptor, which contains enough information to support subsequent point cloud and image feature matching.

[0039] 104. Perform edge consistency analysis and feature matching processing based on the image edge information, feature information and target point cloud data, identify and eliminate the target point cloud data in the occluded area, and obtain optimized target point cloud data.

[0040] By calculating the geometric distance and orientation consistency between the edge points in the point cloud data and the image edge, the matching degree between the point cloud edge and the image edge can be evaluated, thereby identifying inconsistent edge points caused by occlusion, perspective change or noise interference. For feature matching, the similarity measure between feature descriptors is used to establish a correspondence between image feature points and point cloud feature points.

[0041] When identifying occluded areas, we combine the visibility information in the image and the geometric structure of the point cloud data to analyze the spatial relationship between feature points and their association with the image edges. When a point cloud area cannot find the corresponding visual features or edge information in the image, the area is likely to be occluded by other objects. In order to remove these occluded areas, we can use a deep learning-based segmentation algorithm or geometric reasoning method to accurately define the occluded areas and remove them from the target point cloud data.

[0042] The optimized target point cloud data is fused with the camera image to generate color point cloud data. A registration algorithm can be used to align the optimized target point cloud data and the common features in the camera image, map the color information in the camera image to the optimized target point cloud data, and assign a corresponding color value to each point.

[0043] In an embodiment of the present invention, by utilizing the initial point cloud data collected by the radar, the image data collected by the camera, and the internal and external parameter calibration results of the radar and the camera, the initial point cloud data is converted to the camera perspective, and edge consistency analysis and feature matching are performed with the camera image. This can more accurately identify inconsistent areas between the image and the point cloud, thereby effectively eliminating target point cloud data in occluded areas, improving the accuracy of occlusion judgment, and obtaining more optimized and accurate target point cloud data.

[0044] See also Figure 2 Another embodiment of the occlusion analysis method for fusing point cloud and image in the embodiment of the present invention includes:

[0045] 201. Acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera;

[0046] 202. Convert the initial point cloud data according to the calibration result to obtain target point cloud data under the camera perspective;

[0047] The transformation matrix from the radar coordinate system to the camera coordinate system is calculated using the internal and external parameters in the calibration result; the transformation matrix is ​​applied to transform the initial point cloud data into the target point cloud data in the camera coordinate system.

[0048] The camera's intrinsic parameter matrix is ​​extracted from the calibration results. It contains the camera's internal properties such as focal length and optical center coordinates. These parameters define the geometric characteristics of the camera's imaging. Subsequently, the extrinsic parameters obtained by calibration, namely the rotation matrix and translation vector of the radar relative to the camera, are used. These two parameters describe the spatial relationship between the radar coordinate system and the camera coordinate system. By combining the intrinsic parameter matrix with the extrinsic parameter matrix, the complete transformation matrix from the radar coordinate system to the camera coordinate system can be obtained. This transformation matrix is ​​a 4x4 homogeneous transformation matrix that contains all information on rotation, translation, and projection transformations, which can ensure that the point cloud data is accurately transformed from the radar coordinate system to the camera coordinate system.

[0049] When performing the transformation, the homogeneous coordinates of each point are taken as a 4x1 vector, and then the vector is multiplied by the transformation matrix using matrix multiplication. The result of the matrix multiplication is a new 4x1 vector, which represents the homogeneous coordinates of the point in the camera coordinate system. Since the last component of the homogeneous coordinates is usually used to represent the position or distance information of the point, the three-dimensional coordinates of the point in the camera coordinate system can be obtained by ignoring this component. The transformation process is the same for each point in the point cloud data, so it is necessary to traverse all the points in the initial point cloud data, apply the above transformation process to each point, and finally obtain the target point cloud data in the camera coordinate system.

[0050] 203. Perform image segmentation on the camera image, dividing the camera image into multiple regions;

[0051] The specific position and size of each grid unit can be determined by calculating the image size and the grid division parameters, including the number of rows and columns, and then traversing each pixel in the image and assigning it to the corresponding grid unit according to its coordinate position. In this way, each grid unit contains a certain number of pixels, forming an independent image area. The grid-based image segmentation method regards the camera image as a two-dimensional plane and defines a regular grid on this plane. The size and shape of the grid can be adjusted according to actual needs. For example, you can choose to divide the image into rectangular areas of equal size, or divide it non-uniformly according to the characteristics of the image, such as edges, textures, etc. After determining the grid, each grid unit is regarded as an independent area, which covers the entire image space.

[0052] 204. Applying a corresponding edge detection algorithm and feature extraction algorithm to each region to extract edge information and feature points;

[0053] When applying the corresponding edge detection algorithm to each area, algorithms such as Canny, Sobel, Prewitt or Laplacian are usually used. Taking Canny edge detection as an example, the image is first smoothed using a Gaussian filter to reduce the impact of noise on edge detection. Then, the gradient intensity and direction of each pixel are obtained by calculating the first-order derivative of the image. Then, non-maximum suppression is applied to retain only the local maximum in the gradient direction to refine the edge. Finally, double threshold detection is used to determine potential edge points, and these edge points are connected through hysteresis thresholding technology to form a complete edge contour. This process not only retains important edge information in the image, but also reduces false edges caused by noise.

[0054] In terms of feature point extraction, commonly used algorithms include SIFT, SURF and ORB. Taking SIFT as an example, firstly, a scale space pyramid of the image is constructed, and potential key points are detected by comparing the differences of pixel points at different scales. Then, the direction histogram of each key point is calculated to give the key point direction information, thereby enhancing its robustness to rotation. Then, a 128-dimensional feature descriptor is generated through the local gradient information around the key point. These descriptors have a certain invariance to scale, rotation and illumination changes. Finally, these feature descriptors are used to match feature points.

[0055] 205. Integrate the edge information and feature points of each region to obtain image edge information and feature information corresponding to the camera image;

[0056] The edge information and feature point coordinates of all regions are merged, and duplicate or redundant information is removed to ensure that the integrated image edge information and feature information are both accurate and efficient.

[0057] Data structures such as linked lists or arrays can be used to store edge information and feature points of each region for quick access and merging. During the merging process, a deduplication algorithm can be used to identify and remove duplicate or highly similar points by comparing the coordinates, directions, and local gradient information of edge points and feature points, thereby avoiding information redundancy. At the same time, the merged edge contour can be smoothed to eliminate possible breaks or noise points, further improving the continuity and accuracy of edge information. In addition, feature points can be screened and optimized to retain the most representative and stable feature points to improve the efficiency and accuracy of subsequent image matching or recognition.

[0058] The method for smoothing the merged edge contour can be: by using the preset size and shape of the erosion structure element, the redundant pixels on the merged edge contour are gradually stripped off to obtain a preliminary refined edge contour; the pixel points of the preliminary refined edge contour are screened, and based on the domain information and gradient changes of the pixels, the key points that contribute more to the edge contour are retained, and the non-key points that contribute less are removed to obtain a further refined edge contour; the further refined edge contour is contour tracked, and the pixel point sequence of the edge contour is reconstructed through the contour tracking algorithm to ensure the continuity and integrity of the edge contour, and the final edge contour is obtained. The contour tracking algorithm can be a Freeman chain code or a boundary tracking algorithm. Using morphological erosion operations for smoothing can effectively strip off the small noise and redundant pixels on the edge contour, while maintaining the overall shape and position of the edge to a certain extent, thereby obtaining a cleaner, compact and continuous edge contour.

[0059] The method for screening and optimizing feature points can be: sorting and screening feature points according to preset screening criteria, and selecting feature points with moderate description length, high matching stability and strong contrast in the surrounding area. Specifically, first calculate the descriptor length of each feature point, and perform preliminary screening based on the moderation of the descriptor length to remove feature points with too long or too short descriptors, which may contain too much noise or insufficient information; then, evaluate the matching stability of the feature points, and select feature points with small matching errors and high stability by calculating the matching errors of the feature points under different viewing angles, illumination or scale changes; finally, analyze the contrast of the area surrounding the feature points, and select feature points with strong contrast that can be better distinguished from the background.

[0060] 206. Perform edge consistency analysis and feature matching processing based on the image edge information, feature information and target point cloud data, identify and eliminate the target point cloud data in the occluded area, and obtain optimized target point cloud data.

[0061] In the embodiment of the present invention, by comprehensively using technical means such as internal and external parameter calibration of radar and camera, point cloud data conversion, image segmentation, edge detection and feature extraction, edge consistency analysis and feature matching processing, the target point cloud data in the occluded area is accurately identified and eliminated, thereby improving the integrity and accuracy of the optimized target point cloud data.

[0062] See also Figure 3 Another embodiment of the occlusion analysis method for fusing point cloud and image in the embodiment of the present invention includes:

[0063] 301. Acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera;

[0064] 302. Convert the initial point cloud data according to the calibration result to obtain target point cloud data under the camera perspective;

[0065] 303. Perform edge detection and feature extraction processing on the camera image to obtain image edge information and feature information;

[0066] 304. Aligning the image edge information with the target point cloud data;

[0067] Using the internal and external parameter calibration results between the camera and the radar, through a series of transformations and processing, the image edge information is mapped to the target point cloud data in three-dimensional space, or the target point cloud data is converted into a two-dimensional representation consistent with the image edge information perspective, so as to perform comparison and matching in the same coordinate system or perspective. This alignment process ensures the precise correspondence between the image edge information and the target point cloud data in spatial position.

[0068] 305. Perform edge consistency analysis to compare the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identify the area where the edge is discontinuous or misaligned due to occlusion;

[0069] Perform edge extraction on the target point cloud data to form point cloud edge information; align the image edge information with the point cloud edge information; use a geometric consistency evaluation algorithm to calculate the deviation between the image edge and the point cloud edge based on the image edge information and the point cloud edge information; and identify areas with discontinuous or misaligned edges due to occlusion based on the geometric consistency evaluation results.

[0070] The neighborhood search algorithm in the point cloud processing library can be used to find neighboring points around each point in the target point cloud data, and calculate the normal vector difference and curvature change between each point and its neighboring points. When the normal vector difference or curvature change of the point exceeds the preset threshold, the point is considered to be at the edge. By traversing all points, the edge information of the target point cloud data can be extracted.

[0071] The alignment of image edge information and point cloud edge information can be a two-dimensional perspective alignment or a three-dimensional perspective alignment. In the two-dimensional perspective alignment, the image edge information can be projected onto a plane parallel to the camera perspective, and then the projection can be compared with the point cloud edge information that has been processed by perspective transformation and dimensionality reduction. By adjusting the projection parameters and the perspective transformation matrix, the image edge and the point cloud edge can be made to overlap as much as possible on the two-dimensional plane, thereby achieving alignment. In the three-dimensional perspective alignment, the internal and external parameter calibration results of the camera and lidar can be used to convert the image edge points into a three-dimensional space to form a three-dimensional representation of the edge. Then, through the iterative nearest point algorithm or a similar three-dimensional registration method, the best matching relationship between the three-dimensional representation of the image edge and the point cloud edge can be found in the three-dimensional space, and the three-dimensional position of the image edge is continuously adjusted until the two reach the best alignment state in the three-dimensional space.

[0072] For each pair of image edge points and point cloud edge points, the Euclidean distance between them is calculated to measure their proximity in spatial position. At the same time, the normal vector angle of the two edge points is calculated to evaluate their directional consistency. In order to more comprehensively reflect the geometric characteristics of the edge, a shape difference metric of the edge curve is also introduced, which is usually achieved by calculating the local curvature change of the edge point or using high-level features such as shape context. After obtaining these geometric features, they are compared with preset thresholds. For the Euclidean distance, a tolerance range is determined according to the resolution and accuracy of the point cloud data. For the normal vector angle, an angle threshold is set to define the acceptable directional deviation. For the shape difference, a shape similarity threshold is set according to the smoothness and continuity of the edge curve. When the geometric feature values ​​of the edge point exceed these thresholds, the area of ​​edge discontinuity or misalignment caused by occlusion can be identified.

[0073] 306. Matching feature information in the camera image with feature information in the target point cloud data;

[0074] Extract feature points and their corresponding feature descriptors from the feature information in the camera image; determine the feature points and feature descriptors in the target point cloud data, and compare them with the feature descriptors in the camera image; and determine the matching relationship between the feature points in the camera image and the feature points in the target point cloud data based on the comparison results.

[0075] In one implementation, feature descriptors in the camera image can be compared one by one with feature descriptors in the two-dimensional representation of the target point cloud data by using descriptor matching technology, and the descriptor matching technology includes but is not limited to FLANN or brute force matching; finally, based on the similarity between the descriptors, it is determined which feature points are matched in the two data sets, thereby determining the correspondence between the feature points in the camera image and the feature points in the target point cloud data in a two-dimensional perspective, and the similarity between the descriptors can be calculated by Euclidean distance or Hamming distance.

[0076] In one implementation, feature points and their descriptors are extracted from a camera image, and the internal and external parameter calibration results between the camera and the radar are used to map these two-dimensional feature points into a three-dimensional space to form three-dimensional feature points and their corresponding feature descriptors. At the same time, three-dimensional feature points and their feature descriptors are extracted from target point cloud data, and a three-dimensional feature descriptor matching algorithm is used to compare the three-dimensional feature descriptors mapped from the camera image with the three-dimensional feature descriptors in the target point cloud data. The three-dimensional feature descriptor matching algorithm includes but is not limited to a nearest neighbor search based on a KD tree, and determines which three-dimensional feature points are matched between the camera image and the target point cloud data based on the similarity between the descriptors, thereby establishing a correspondence between the two.

[0077] 307. According to the feature matching results and the edge consistency analysis results, identify and mark the target point cloud data of the occluded area;

[0078] Using the preset occlusion area recognition model, each point in the target point cloud data is classified and judged; the target point cloud data classified as the occlusion area is marked for subsequent removal processing.

[0079] The occluded area recognition model classifies and judges each point in the target point cloud data. The specific method is: based on the feature matching results, when it is found that the feature points in the target data point cloud are inconsistent with the feature points in the camera image during the matching process, and these inconsistent feature points happen to be located in the area previously determined by edge consistency analysis, it is considered that the point cloud data corresponding to these feature points are occluded, and accordingly these occluded feature points and the occluded areas formed by them are marked in the target point cloud data.

[0080] 308. Eliminate the target point cloud data marked as the occluded area to obtain optimized target point cloud data.

[0081] The occluded areas are classified according to their shape, size and position information; different culling strategies are matched to different types of occluded areas; and based on each culling strategy, the target point cloud data of each occluded area is culled to obtain optimized target point cloud data.

[0082] By calculating the boundary points of the occluded area and fitting its geometric shape, the occluded area can be roughly divided into two categories: regular shape and irregular shape. Occluded areas with regular shapes are usually easier to identify and remove because they have clear boundaries and predictable shape characteristics; for occluded areas with irregular shapes, it is necessary to further analyze the complexity and irregularity of their boundaries, as well as the relative position relationship with other occluded areas or target point cloud data; in addition, the size of the occluded area is also an important classification basis, which can help determine the strength and scope of the removal strategy. For example, for occluded areas with smaller areas, these points can be directly deleted to simplify processing, while for occluded areas with larger areas, more complex removal algorithms or methods may be required to ensure that the removal operation does not damage the structure and integrity of the entire point cloud data.

[0083] In the embodiment of the present invention, by fusing the point cloud data collected by the radar with the camera image, converting it using the internal and external parameter calibration results, and performing edge consistency analysis and feature matching, the occluded area can be accurately identified, thereby effectively eliminating the point cloud data in the occluded area, thereby improving the accuracy and completeness of the optimized target point cloud data.

[0084] The above describes the occlusion analysis method for fusing point cloud and image in the embodiment of the present invention. The following describes the occlusion analysis device for fusing point cloud and image in the embodiment of the present invention. Figure 4 In one embodiment of the present invention, an occlusion analysis device for fusing point cloud and image includes:

[0085] An acquisition module 401 is used to acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera;

[0086] The processing module 402 is used to convert the initial point cloud data according to the calibration result to obtain the target point cloud data under the camera perspective;

[0087] Extraction module 403, used to perform edge detection and feature extraction processing on the camera image to obtain image edge information and feature information;

[0088] The elimination module 404 is used to perform edge consistency analysis and feature matching processing based on the image edge information, feature information and target point cloud data, identify and eliminate the target point cloud data in the occluded area, and obtain optimized target point cloud data.

[0089] In an embodiment of the present invention, by utilizing the initial point cloud data collected by the radar, the image data collected by the camera, and the internal and external parameter calibration results of the radar and the camera, the initial point cloud data is converted to the camera perspective, and edge consistency analysis and feature matching are performed with the camera image. This can more accurately identify inconsistent areas between the image and the point cloud, thereby effectively eliminating target point cloud data in occluded areas, improving the accuracy of occlusion judgment, and obtaining more optimized and accurate target point cloud data.

[0090] See also Figure 5 Another embodiment of the occlusion analysis device for fusing point cloud and image in the embodiment of the present invention includes:

[0091] An acquisition module 401 is used to acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera;

[0092] The processing module 402 is used to convert the initial point cloud data according to the calibration result to obtain the target point cloud data under the camera perspective;

[0093] Extraction module 403, used to perform edge detection and feature extraction processing on the camera image to obtain image edge information and feature information;

[0094] The elimination module 404 is used to perform edge consistency analysis and feature matching processing based on the image edge information, feature information and target point cloud data, identify and eliminate the target point cloud data in the occluded area, and obtain optimized target point cloud data.

[0095] Optionally, the processing module 402 may be specifically configured to:

[0096] The transformation matrix from the radar coordinate system to the camera coordinate system is calculated using the internal and external parameters in the calibration result; the transformation matrix is ​​applied to transform the initial point cloud data into the target point cloud data in the camera coordinate system.

[0097] Optionally, the extraction module 403 includes:

[0098] The segmentation unit 4031 is used to perform image segmentation on the camera image and divide the camera image into multiple regions;

[0099] An extraction unit 4032 is used to apply a corresponding edge detection algorithm and feature extraction algorithm to each region to extract edge information and feature points;

[0100] The integration unit 4033 is used to integrate the edge information and feature points of each area to obtain the image edge information and feature information corresponding to the camera image.

[0101] Optionally, the extraction unit 4032 is specifically used for:

[0102] According to the content, structure or application scenario characteristics of each region, select and apply the corresponding edge detection algorithm to extract the edge information of each region; according to the image characteristics contained in each region or the target to be identified, select and apply the corresponding feature extraction algorithm to extract the feature points of each region.

[0103] Optionally, the elimination module 404 includes:

[0104] An alignment unit 4041 is used to align image edge information with target point cloud data;

[0105] An analysis unit 4042 is used to perform edge consistency analysis, compare the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identify areas where the edge is discontinuous or misaligned due to occlusion;

[0106] A matching unit 4043, used to match feature information in the camera image with feature information in the target point cloud data;

[0107] The marking unit 4044 is used to identify and mark the target point cloud data of the occluded area according to the feature matching result and the edge consistency analysis result;

[0108] The elimination unit 4045 is used to eliminate the target point cloud data marked as the occlusion area to obtain the optimized target point cloud data.

[0109] Optionally, the analysis unit 4042 is specifically used for:

[0110] Perform edge extraction on the target point cloud data to form point cloud edge information; align the image edge information with the point cloud edge information; use a geometric consistency evaluation algorithm to calculate the deviation between the image edge and the point cloud edge based on the image edge information and the point cloud edge information; and identify areas with discontinuous or misaligned edges due to occlusion based on the geometric consistency evaluation results.

[0111] Optionally, the elimination unit 4045 is specifically used for:

[0112] The occluded areas are classified according to their shape, size and position information; different culling strategies are matched to different types of occluded areas; and based on each culling strategy, the target point cloud data of each occluded area is culled to obtain optimized target point cloud data.

[0113] In the embodiment of the present invention, by comprehensively using technical means such as internal and external parameter calibration of radar and camera, point cloud data conversion, image segmentation, edge detection and feature extraction, edge consistency analysis and feature matching processing, the target point cloud data in the occluded area is accurately identified and eliminated, thereby improving the integrity and accuracy of the optimized target point cloud data.

[0114] above Figure 4 and Figure 5 The occlusion analysis device for fusing point cloud and image in the embodiment of the present invention is described in detail from the perspective of modular functional entity. The occlusion analysis device for fusing point cloud and image in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0115] See also Figure 6 As shown, the occlusion analysis device for fusing point cloud and image includes a processor 600 and a memory 601. The memory 601 stores machine executable instructions that can be executed by the processor 600. The processor 600 executes the machine executable instructions to implement the above-mentioned occlusion analysis method for fusing point cloud and image.

[0116] Further, Figure 6 The occlusion analysis device for fusing point cloud and image shown in the figure further includes a bus 602 and a communication interface 603 , and the processor 600 , the communication interface 603 and the memory 601 are connected via the bus 602 .

[0117] Among them, the memory 601 may include a high-speed random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), for example, at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 603 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 602 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0118] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 600. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601 , and the processor 600 reads the information in the memory 601 and completes the method steps of the above-mentioned embodiment in combination with its hardware.

[0119] The present invention also provides an occlusion analysis device for fusing point clouds and images, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the occlusion analysis method for fusing point clouds and images in the above-mentioned embodiments. The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the steps of the occlusion analysis method for fusing point clouds and images.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0122] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An occlusion analysis method for fusing point cloud and image, characterized in that: The occlusion analysis method for fusing point cloud and image includes: Acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera; The initial point cloud data is converted according to the calibration result to obtain target point cloud data under the camera perspective; Performing edge detection and feature extraction processing on the camera image to obtain image edge information and feature information; Perform edge consistency analysis and feature matching processing according to the image edge information, the feature information and the target point cloud data, identify and eliminate the target point cloud data in the occluded area, and obtain optimized target point cloud data; The method includes: performing edge consistency analysis and feature matching processing according to the image edge information, the feature information and the target point cloud data, identifying and eliminating the target point cloud data in the occluded area, and obtaining the optimized target point cloud data, including: aligning the image edge information with the target point cloud data; performing edge consistency analysis, comparing the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identifying the area where the edge is discontinuous or misaligned due to occlusion; matching the feature information in the camera image with the feature information in the target point cloud data; identifying and marking the target point cloud data in the occluded area according to the feature matching results and the edge consistency analysis results; and eliminating the target point cloud data marked as the occluded area to obtain the optimized target point cloud data.

2. The occlusion analysis method for integrating point cloud and image according to claim 1, characterized in that: The converting and processing the initial point cloud data according to the calibration result to obtain target point cloud data under the camera perspective includes: Calculating a transformation matrix from a radar coordinate system to a camera coordinate system using the internal and external parameters in the calibration result; The initial point cloud data is converted into target point cloud data in a camera coordinate system by applying the conversion matrix.

3. The occlusion analysis method for integrating point cloud and image according to claim 1, characterized in that: The performing edge detection and feature extraction processing on the camera image to obtain image edge information and feature information includes: Performing image segmentation on the camera image to divide the camera image into multiple regions; Apply the corresponding edge detection algorithm and feature extraction algorithm to each area to extract edge information and feature points; The edge information and feature points of each region are integrated to obtain the image edge information and feature information corresponding to the camera image.

4. The occlusion analysis method for integrating point cloud and image according to claim 3, characterized in that: The step of applying a corresponding edge detection algorithm and feature extraction algorithm to each region to extract image edge information and feature points includes: According to the characteristics of the content, structure or application scenario of each region, select and apply the corresponding edge detection algorithm to extract the edge information of each region; According to the image characteristics contained in each area or the target to be identified, the corresponding feature extraction algorithm is selected and applied to extract the feature points of each area.

5. The occlusion analysis method for integrating point cloud and image according to claim 1, characterized in that: The performing edge consistency analysis, comparing the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identifying the area where the edge is discontinuous or misaligned due to occlusion, includes: Performing edge extraction on the target point cloud data to form point cloud edge information; Aligning the image edge information with the point cloud edge information; Using a geometric consistency evaluation algorithm, based on the image edge information and the point cloud edge information, the deviation between the image edge and the point cloud edge is calculated; Based on the geometric consistency evaluation results, the regions with edge discontinuities or misalignments due to occlusion are identified.

6. The occlusion analysis method for integrating point cloud and image according to claim 1, characterized in that: The step of removing the target point cloud data marked as an occluded area to obtain optimized target point cloud data includes: Classifying the occluded area according to the shape, size and position information of the occluded area; Match different culling strategies to different types of occluded areas; Based on each elimination strategy, the target point cloud data of each occluded area is eliminated to obtain optimized target point cloud data.

7. An occlusion analysis device for integrating point cloud and image, characterized in that: The occlusion analysis device for fusing point cloud and image comprises: An acquisition module, used to acquire initial point cloud data and camera images, as well as internal and external parameter calibration results of the radar and the camera, wherein the initial point cloud data is point cloud data of the target environment collected by the radar, and the camera image is image data of the target environment collected by the camera; A processing module, used for converting the initial point cloud data according to the calibration result to obtain target point cloud data under the camera perspective; An extraction module, used to perform edge detection and feature extraction processing on the camera image to obtain image edge information and feature information; A removal module is used to perform edge consistency analysis and feature matching processing according to the image edge information, the feature information and the target point cloud data, identify and remove the target point cloud data in the occluded area, and obtain optimized target point cloud data; The elimination module includes: an alignment unit, which is used to align the image edge information with the target point cloud data; an analysis unit, which is used to perform edge consistency analysis, compare the geometric consistency of the image edge information with the corresponding edge in the target point cloud data, and identify the area with discontinuous or misaligned edges caused by occlusion; a matching unit, which is used to match the feature information in the camera image with the feature information in the target point cloud data; a marking unit, which is used to identify and mark the target point cloud data in the occluded area according to the feature matching results and the edge consistency analysis results; and a elimination unit, which is used to eliminate the target point cloud data marked as the occluded area to obtain optimized target point cloud data.

8. The occlusion analysis device for integrating point cloud and image according to claim 7, characterized in that: The extraction module includes: a segmentation unit, which is used to perform image segmentation on the camera image and divide the camera image into multiple areas; an extraction unit, which is used to apply the corresponding edge detection algorithm and feature extraction algorithm to each area to extract edge information and feature points; and an integration unit, which is used to integrate the edge information and feature points of each area to obtain the image edge information and feature information corresponding to the camera image.

9. An occlusion analysis device for fusing point cloud and image, characterized in that: The occlusion analysis device for fusing point cloud and image comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instruction in the memory so that the occlusion analysis device for fusing point cloud and image executes the occlusion analysis method for fusing point cloud and image as described in any one of claims 1-6.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by the processor, the occlusion analysis method for fusing point cloud and image as described in any one of claims 1 to 6 is implemented.

Images