A method and device for generating pseudo points based on mixed light images

Abstract

The present application provides a method and device for generating pseudo points based on mixed light images. The method provided by the present application includes: depth completion of the first depth map and the second depth map, generation of the third depth map and the fourth depth map, and calculation of the multimodal geometric points corresponding to the third depth map and the fourth depth map; wherein the first depth map and the third depth map are infrared data, and the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map; the multimodal geometric points and the lidar geometric points are input into the encoder to generate multiple sparse cost volumes; the sparse cost volumes are regularized to generate target confidence and target offset; the third depth map and the fourth depth map are corrected based on the target confidence and the target offset to generate target pseudo points. The method and device for generating pseudo points based on mixed light images provided by the present application are used to generate high-precision and high-quality pseudo points.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for generating pseudo points based on mixed light images. Background Art

[0002] Environmental perception is an important part of object detection, which requires the system to accurately identify objects in the surrounding environment and estimate their position, size and shape. Although existing pseudo point generation technologies have their own advantages, they often find it difficult to strike a balance between accuracy, robustness and efficiency. Especially in all-weather conditions, how to generate high-quality pseudo points has become an urgent problem to be solved.

[0003] The existing pseudo-point generation method is mainly based on monocular visible light cameras. Although this method is efficient, its accuracy is poor due to the pathological problem of estimating high-dimensional information from low-dimensional information. Therefore, researchers use pseudo-point generation methods based on binocular visible light cameras to generate pseudo-points. Binocular visible light cameras improve accuracy, but have limited spatial resolution and are difficult to maintain scene details. The performance of these methods often drops significantly when faced with extreme lighting conditions and cannot meet the needs of actual use. Summary of the invention

[0004] In view of this, the present application provides a method and device for generating pseudo points based on mixed light images, so as to generate high-precision and high-quality pseudo points.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] In a first aspect, the present application provides a pseudo point generation method based on a mixed light image, the method comprising:

[0007] Depth-complete the first depth map and the second depth map to generate a third depth map and a fourth depth map, and calculate multimodal geometric points corresponding to the third depth map and the fourth depth map;

[0008] The first depth map and the third depth map are infrared data, and the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map;

[0009] Inputting the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes, each of the sparse cost volumes comprising at least two spectral data, and the spectral data of each sparse cost volume having a different mixing ratio;

[0010] Regularizing the sparse cost volume, and generating target confidences and target offsets corresponding to the third depth map and the fourth depth map based on some of the sparse cost volumes in the multiple sparse cost volumes;

[0011] The third depth map and the fourth depth map are corrected based on the target confidence and the target offset to generate a target pseudo point, wherein the third depth map and the fourth depth map correspond to different partial sparse cost volumes.

[0012] The second aspect of the present application provides a pseudo point generation device based on a mixed light image, the device comprising a processing module, a generation module and a correction module; wherein:

[0013] The processing module is used to deeply complete the first depth map and the second depth map, generate a third depth map and a fourth depth map, and calculate multimodal geometric points corresponding to the third depth map and the fourth depth map;

[0014] The first depth map and the third depth map are infrared data, and the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map;

[0015] The generating module is used to input the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes; wherein each of the sparse cost volumes includes at least two spectral data, and the spectral data mixing ratios of the sparse cost volumes are different;

[0016] The generating module is used to regularize the sparse cost volume, and generate the target confidence and target offset corresponding to the third depth map and the fourth depth map based on part of the sparse cost volumes in the multiple sparse cost volumes respectively;

[0017] The correction module is used to correct the third depth map and the fourth depth map based on the target confidence and the target offset to generate a target pseudo point, wherein the third depth map and the fourth depth map correspond to different partial sparse cost volumes.

[0018] The pseudo-point generation method and device based on mixed light images provided in the present application first deeply complete the first depth map and the second depth map to generate the third depth map and the fourth depth map, calculate the multimodal geometric points corresponding to the third depth map and the fourth depth map, and then input the multimodal geometric points and the laser radar geometric points into the encoder to generate multiple sparse cost bodies, and then regularize the sparse cost bodies to generate target confidence and target offset, and finally correct the third depth map and the fourth depth map based on the target confidence and target offset to generate target pseudo points. In this way, by completing the first depth map and the second depth map with sparse data volume, the third depth map and the fourth depth map with dense data volume can be obtained. The method provided by the present invention comprehensively uses visible light data, infrared data and laser radar data to obtain the depth map with the most comprehensive information and accurately generate pseudo points. Specifically, since visible light, infrared data and lidar data are in different coordinate systems, it is difficult to directly fuse them. Therefore, the present invention first obtains depth maps after visible light and infrared data are completed by parallel processing of visible light and infrared light data, thereby improving the density of the depth map, changing the sparse depth map into a large-grained depth map with a density greater than that of the first level, and completing the first completion of the depth map information; then, in order to utilize the fusion information between the three, the three spectral data are fused to the same coordinate system using the cost body, and different types of cost bodies are generated according to the type of spectral data to be supplemented, and a cost body that fuses the three spectral data but has different information content and composition is constructed. Finally, according to the need for missing large-grained depth map information, the corresponding cost body is selected to perform a second completion on the depth map after the first completion. In this way, the rich data can accurately obtain depth information under complex lighting conditions, ensuring the accuracy and reliability of subsequent pseudo-point generation, and fully utilizing the information in a single spectral data while comprehensively utilizing the fusion information of the three spectral data. Furthermore, by calculating the sparse cost volume through multimodal geometric points and lidar geometric points, since multimodal geometric points and lidar geometric points only contain position information, the efficiency of calculation can be effectively improved and the computing resources can be reduced. Furthermore, the sparse cost volume can be combined with the high-resolution features of infrared, visible light and lidar to determine the consistency between multimodal sensors, thereby ensuring the reliability of subsequent pseudo-point generation. Furthermore, by jointly obtaining accurate confidence to judge the reliability of the data in the third depth map and the fourth depth map through multiple sensors, and completing calibration fine-tuning based on the target offset, the reliability and accuracy of the target pseudo-point can be ensured. In this way, by integrating data from multiple sensors, it is possible to efficiently generate high-quality pseudo-points in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a pseudo point generation method based on a mixed light image provided by an exemplary embodiment of the present application;

[0020] Figure 2 A schematic diagram of a pseudo point generation model based on a mixed light image provided by an exemplary embodiment of the present application;

[0021] Figure 3 A schematic diagram of a target pseudo point provided for an exemplary embodiment of the present application;

[0022] Figure 4 A hardware structure diagram of a pseudo point generating device based on a mixed light image where the pseudo point generating device based on a mixed light image of the present application is located;

[0023] Figure 5 This is a structural schematic diagram of Embodiment 1 of a pseudo point generating device based on a mixed light image provided in the present application. DETAILED DESCRIPTION

[0024] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0025] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0026] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0027] The present application provides a method and device for generating pseudo points based on mixed light images, so as to generate high-precision and high-quality pseudo points.

[0028] The pseudo-point generation method and device based on mixed light images provided in the present application first deeply complete the first depth map and the second depth map to generate the third depth map and the fourth depth map, calculate the multimodal geometric points corresponding to the third depth map and the fourth depth map, and then input the multimodal geometric points and the laser radar geometric points into the encoder to generate multiple sparse cost bodies, and then regularize the sparse cost bodies to generate target confidence and target offset, and finally correct the third depth map and the fourth depth map based on the target confidence and target offset to generate target pseudo points. In this way, by completing the first depth map and the second depth map with sparse data volume, the third depth map and the fourth depth map with dense data volume can be obtained. The method provided by the present invention comprehensively uses visible light data, infrared data and laser radar data to obtain the depth map with the most comprehensive information volume and accurately generate pseudo points. Specifically, since visible light, infrared data and laser radar data are in different coordinate systems, it is difficult to directly fuse them. Therefore, the present invention first obtains the depth map after the visible light and infrared data are supplemented by parallel processing of visible light and infrared data, thereby improving the density of the depth map, changing the sparse depth map to a large-grained depth map with a density greater than the first level, and completing the first supplement of the depth map information; then, in order to utilize the fusion information between the three, the cost body is used to fuse the three spectral data into the same coordinate system, and different types of cost bodies are generated according to the type of spectral data to be supplemented, and a cost body that fuses the three spectral data but has different information content and composition is constructed. Finally, according to the need for missing information of the large-grained depth map, the corresponding cost body is selected to perform the second supplement of the depth map after the first supplement. In this way, the depth information can be accurately obtained under complex lighting conditions through rich data, ensuring the accuracy and reliability of subsequent pseudo-point generation. Further, the sparse cost body is calculated by multimodal geometric points and laser radar geometric points. Since the multimodal geometric points and laser radar geometric points only contain position information, the efficiency of the calculation can be effectively improved and the resources of the calculation can be reduced. Furthermore, the sparse cost volume can be combined with the high-resolution features of infrared, visible light and lidar to determine the consistency between multimodal sensors, ensuring the reliability of subsequent pseudo-point generation. Furthermore, the reliability of the data in the third depth map and the fourth depth map is determined by combining multiple sensors to obtain accurate confidence, and calibration fine-tuning is completed based on the target offset, ensuring the reliability and accuracy of the target pseudo-point. In this way, by integrating multiple sensor data, it is possible to efficiently generate high-quality pseudo-points in complex environments.

[0029] Specific embodiments are given below to introduce the technical solution of the present application in detail.

[0030] Figure 1 This is a flowchart of a pseudo point generation method based on a mixed light image provided by an exemplary embodiment of the present application. Figure 1, the method provided in this embodiment may include:

[0031] S101, depth-complete a first depth map and a second depth map to generate a third depth map and a fourth depth map, and calculate multimodal geometric points corresponding to the third depth map and the fourth depth map.

[0032] The first depth map and the third depth map are infrared data, the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map.

[0033] Specifically, the first depth map and the second depth map are sparse depth maps, which contain a limited number of depth measurement points. In the first depth map and the second depth map, most of the pixels have no corresponding depth values.

[0034] It should be noted that the first depth map and the second depth map have the characteristics of high precision but low data density. In other words, the depth value of each measurement point is very accurate, but only a small area of ​​the entire scene is measured.

[0035] In a specific implementation, the depth measurement points in the first depth map come from an infrared detector, and the depth measurement points in the second depth map come from a visible light detector.

[0036] Furthermore, the infrared detector works in the infrared band and is a sensor that senses and measures the infrared radiation of an object. It should be noted that the infrared detector is sensitive to temperature and can capture the changes in infrared radiation caused by the temperature difference of the object. In the case of poor lighting conditions, the infrared detector can still work effectively and provide reliable object contour and position information.

[0037] Furthermore, a visible light detector operates in the visible light band and is a sensor that captures and records light within the visible spectrum. It should be noted that a visible light detector can capture rich color information and texture details, providing an intuitive and easy-to-understand image for environmental perception.

[0038] Furthermore, the first depth map and the second depth map are sparse, only containing the depth values ​​of some points in the scene, while the depths of most other areas are unknown, and depth completion processing is required to generate dense and complete depth maps.

[0039] In specific implementation, the first depth map is depth-completed to obtain the third depth map, and the second depth map is depth-completed to obtain the fourth depth map. It can be understood that the first depth map and the third depth map are both infrared data, and the second depth map is depth-completed to obtain visible light data, and the two completion processes are independent of each other.

[0040] Furthermore, the first depth map and the second depth map may be depth-completed based on a traditional depth completion method, or may be depth-completed based on a neural network depth completion method, which is not limited in the present embodiment.

[0041] A specific embodiment is given below to introduce the acquisition process of the third depth map and the fourth depth map in detail:

[0042] (1) Inputting the infrared image and the first depth map into a dual-spectrum depth completion module, and acquiring multi-scale infrared features of the infrared image based on the dual-spectrum depth completion module.

[0043] Specifically, the infrared image is an image captured by an infrared detector.

[0044] Furthermore, Figure 2 A schematic diagram of a pseudo point generation model based on a mixed light image provided by an exemplary embodiment of the present application. Figure 2 , the infrared image and the first depth map are input into the dual-spectrum completion module, in which the input infrared image and the first depth map are processed by the residual encoder-decoder structure to extract multi-scale infrared features. These features contain not only the texture and color information of the image, but also the structural information of the sparse depth map.

[0045] (2) Inputting the visible light image and the second depth map into a dual-spectrum depth completion module, and acquiring multi-scale visible light features of the visible light image based on the dual-spectrum depth completion module.

[0046] Specifically, the visible light image is an image captured by a visible light detector.

[0047] For further information, please refer to Figure 2 , the visible light image and the second depth map are input into the dual-spectrum completion module, in which the input visible light image and the second depth map are processed by the residual encoder-decoder structure to extract multi-scale visible light features. These features contain not only the texture and color information of the image, but also the structural information of the sparse depth map.

[0048] (3) Completing the first depth map based on the multi-scale infrared features to generate the third depth map, and completing the second depth map based on the multi-scale visible light features to generate the fourth depth map.

[0049] Specifically, using the extracted multi-scale features, the dual-spectrum completion module predicts the depth distribution probability corresponding to each pixel. The depth distribution probability obtained through neural network learning represents the possible depth value of each pixel and its corresponding probability.

[0050] Furthermore, the dual-spectrum completion module converts the corresponding depth distribution probability into a specific depth value based on the multi-scale infrared features, and predicts and generates a third depth map.

[0051] Similarly, the dual-spectrum completion module converts the corresponding depth distribution probability into a specific depth value based on the multi-scale visible light features, and predicts and generates a fourth depth map.

[0052] The pseudo point generation method based on mixed light image provided in this embodiment first inputs the infrared image and the first depth map into the dual-spectrum depth completion module to obtain multi-scale infrared features, and inputs the visible light image and the second depth map into the dual-spectrum depth completion module to obtain multi-scale visible light features, and then generates a third depth map based on most multi-scale infrared features, and generates a fourth depth map based on most multi-scale visible light features. In this way, through the dual-spectrum depth completion module, the depth value of the unknown area in the scene can be accurately inferred by combining the features of the infrared image and the visible light image, thereby ensuring the accuracy of the third depth map and the fourth depth map. Furthermore, the infrared image and the visible light image are complementary in terms of the amount of information. The infrared image mainly captures the thermal radiation information of the object, is sensitive to temperature, but not to lighting conditions; while the visible light image contains rich texture, color and edge information. In this way, the two types of data are processed separately by the dual-spectrum depth completion module, thereby achieving maximum utilization of information and generating more comprehensive and accurate third depth maps and fourth depth maps.

[0053] Furthermore, the third depth map and the fourth depth map are generated by the dual-spectral depth completion module, which represent the possible depth values ​​of each pixel position in the scene. In three-dimensional space, the multimodal geometric points in the third depth map and the fourth depth map are points obtained by converting each pixel position and its corresponding depth value into a three-dimensional space coordinate system, and the three-dimensional structural information of the scene can be represented by the multimodal geometric points.

[0054] Furthermore, multimodal geometric points can be obtained from the third depth map and the fourth depth map based on a traditional method of obtaining geometric points, or multimodal geometric points can be obtained from the third depth map and the fourth depth map based on a method of obtaining geometric points of a neural network. This is not limited to the present embodiment.

[0055] A specific embodiment is given below to introduce the process of obtaining multi-modal geometric points in detail:

[0056] (1) Identify the scene to be identified based on the laser radar and generate a radar coordinate system.

[0057] Specifically, the LiDAR detector determines the distance of the target object by emitting laser pulses and measuring the time difference between them being reflected from the target. In specific implementation, the LiDAR detector includes a laser transmitter and a receiver. The laser transmitter emits short and intense laser pulses to the target. When these pulses hit the surface of the target object, they are reflected. The reflected light pulses are captured by the receiver. By calculating the time required for the laser pulse to travel back and forth, and combining it with the speed of light, the distance between the target and the LiDAR detector can be accurately calculated.

[0058] Furthermore, a radar coordinate system is generated according to the position of the laser radar detector. In specific implementation, a radar coordinate system centered on the laser radar is generated according to the calibration parameters of the laser radar detector. It should be noted that the calibration parameters of the laser radar detector may include position information, attitude information, scanning range information, etc. The radar coordinate system can be used to unify the spatial positions of different sensor data in subsequent processing.

[0059] (2) Back-projecting the third depth map and the fourth depth map into the radar coordinate system to obtain the multimodal geometric point.

[0060] In a specific implementation, each pixel representing the depth in the third depth map and the fourth depth map is combined with its position in the image, and these two-dimensional pixel points and their depth values ​​are back-projected into the radar coordinate system through the internal parameters such as the focal length and optical center of the infrared detector and the visible light detector and the external parameters such as the rotation matrix and translation vector between the infrared detector, the visible light detector and the laser radar detector to obtain multimodal geometric points. For example, in one embodiment, the depth value d corresponding to each pixel point (u, v) is converted to the radar coordinate system through the transformation matrix between the infrared detector, the visible light detector and the laser radar to obtain a multimodal geometric point.

[0061] The pseudo-point generation method based on mixed light images provided in this embodiment first generates a radar coordinate system based on the laser radar detection and identification of the scene to be identified, and then back-projects the third depth map and the fourth depth map into the radar coordinate system to obtain multimodal geometric points. In this way, by back-projecting the third depth map and the fourth depth map into a unified radar coordinate system, a high degree of consistency in the spatial position of different sensor data is achieved, which facilitates the subsequent generation of pseudo-points. Furthermore, infrared images can still provide clear thermal radiation information when the lighting conditions are poor, while visible light images contain rich texture and color information. After converting the third depth map and the fourth depth map into multimodal geometric points, their respective advantages can be fully utilized to achieve information complementarity, thereby improving the overall three-dimensional perception capability. Furthermore, the multimodal geometric points can effectively reduce the amount of calculation in the operation process, and can greatly improve the efficiency of pseudo-point generation.

[0062] S102: Input the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes.

[0063] Specifically, the laser radar geometric point is a geometric point obtained when the laser radar detector detects the scene to be identified. The laser radar geometric point can be used to represent the position information of the object in the scene to be identified.

[0064] Furthermore, the sparse cost geometry is used to store and represent the matching cost between corresponding pixels or feature points in two or more views. Each of the sparse cost volumes includes at least two spectral data, and the spectral data mixing ratios of each of the sparse cost volumes are different. It should be noted that the sparse cost geometry only stores the cost values ​​at those positions that have significant significance or potential matching possibilities in practical applications, thereby significantly reducing memory consumption and improving computational efficiency.

[0065] A specific embodiment is given below to explain in detail the process of obtaining the sparse cost volume:

[0066] (1) Extracting features of the multimodal geometric points and the laser radar geometric points to obtain target features; wherein the target features include infrared features, visible light features, and laser radar features.

[0067] Specifically, feature extraction can be performed on the multimodal geometric points and the laser radar geometric points based on a traditional feature extraction method, or based on a neural network feature extraction method, which is not limited in this embodiment. For example, in one embodiment, infrared features, visible light features, and laser radar features can be extracted from the geometric points through a convolutional neural network CNN.

[0068] (2) Acquire valid voxels and voxel ratios based on the target features, process coordinates corresponding to the valid voxels based on the voxel ratios, and generate a sparse cost volume.

[0069] Specifically, valid voxels are three-dimensional data points or regions that exist in the scene to be identified and have practical significance, and these points or regions are related to subsequent data processing and analysis.

[0070] In the specific implementation, by matching and verifying the target features, we obtain points that contribute significantly to the description of the scene to be identified, and determine these points as valid voxels. It should be noted that valid voxels not only contain position information, but may also contain a variety of feature information such as color, texture, and reflection intensity, which is an important basis for subsequent tasks such as 3D reconstruction, depth estimation, or object detection.

[0071] For further information, please refer to Figure 2, multimodal geometric points and lidar geometric points can be input into the sparse cost volume module, and a sparse cost volume can be generated through the sparse cost volume module.

[0072] The pseudo-point generation method based on mixed light images provided in this embodiment first performs feature extraction on multimodal geometric points and lidar geometric points to obtain target features, and then obtains effective voxels and voxel ratios based on the target features, processes the coordinates corresponding to the effective voxels based on the voxel ratio, and generates a sparse cost volume. In this way, by using a sparse cost volume instead of a traditional full-density cost volume, the amount of calculation and memory consumption are greatly reduced. Furthermore, since only effective voxels and their surrounding areas are processed, unnecessary calculations are avoided, thereby improving the overall processing speed. Furthermore, through precise feature extraction and the construction of a sparse cost volume, the consistency between different sensor data can be better quantified, providing powerful guidance for subsequent pseudo-point generation, and helping to improve the accuracy of pseudo-point generation.

[0073] A specific embodiment is given below to introduce the generation process of the sparse cost volume in detail:

[0074] Step 1: Pair the infrared feature, the visible light feature and the lidar feature in sequence based on the voxel ratio to obtain a plurality of multimodal feature pairs.

[0075] Specifically, when processing features of different modalities, features of other modalities are used for pairing to obtain multi-modal feature pairs that are combined in pairs.

[0076] It should be noted that the ratios of different modes in a multimodal feature pair are different. In specific implementation, the specific ratio of the two modes in a multimodal feature pair is determined according to actual needs, and this is not limited in this embodiment. For example, in one embodiment, when generating a multimodal feature pair corresponding to an infrared feature, the ratio of the laser radar feature and the visible light feature is set to be lower than the ratio of the infrared feature.

[0077] Optionally, the infrared feature is corrected based on the visible light feature and the laser radar feature to obtain the multimodal feature pair corresponding to the infrared feature; wherein, in the multimodal feature pair corresponding to the infrared feature, the infrared feature is the main body;

[0078] The visible light feature is corrected based on the infrared feature and the laser radar feature to obtain the multimodal feature pair corresponding to the visible light feature; wherein, in the multimodal feature pair corresponding to the visible light feature, the visible light feature is the main body.

[0079] In specific implementation, infrared features can be used as the main body to generate multimodal feature pairs of infrared features and visible light features, as well as multimodal feature pairs of infrared features and radar features. Visible light features can also be used as the main body to generate multimodal feature pairs of visible light features and infrared features, as well as multimodal feature pairs of visible light features and radar features.

[0080] The step of sequentially pairing the infrared feature, the visible light feature, and the laser radar feature based on the voxel ratio to obtain a plurality of multimodal feature pairs includes:

[0081] Determine the spectrum type to be supplemented according to the spectrum type of the depth map to be corrected; determine the sparse cost volume corresponding to the depth map to be corrected based on the spectrum type to be supplemented, and one sparse cost volume corresponds to one spectrum type to be supplemented; determine the voxel ratio of each spectrum in each sparse cost volume corresponding to the depth map to be corrected based on all spectrum types of the depth map to be corrected; wherein the sparse cost volume corresponding to each depth map to be corrected includes spectrum information of all spectrum types to be supplemented, and the spectrum information of the maximum voxel ratio of the sparse cost volume corresponding to each depth map to be corrected is different. Specifically, the depth map to be corrected includes the third depth map and the fourth depth map. If it is the third depth map, the spectrum information composed of it includes infrared spectrum information and visible spectrum information. Therefore, the corresponding spectrum type to be supplemented includes infrared light and visible light, so it needs two sparse cost volumes, both of which include infrared light information and visible light information, but the infrared light information accounts for the highest proportion in one sparse cost volume, and the visible light information accounts for the highest proportion in the other sparse cost volume.

[0082] Step 2: Calculate the similarity of the multimodal feature pairs to generate the valid voxels.

[0083] Specifically, the similarity of the multimodal feature pairs can be calculated based on a traditional similarity calculation method, or based on a neural network similarity calculation method, which is not limited in this embodiment. For example, in one embodiment, the Euclidean distance or cosine similarity method can be used to generate valid voxels.

[0084] Furthermore, multimodal features with similarity greater than a preset similarity may be determined as valid voxels. It is understandable that multimodal features with large similarity are consistent in different sensor data and can reflect the real situation of the scene to be identified, so they are regarded as valid voxels.

[0085] Step 3: Generate the sparse cost volume based on the coordinates of the valid voxels.

[0086] In the specific implementation, in this step, an empty sparse cost volume is created, and its structure should be able to accommodate all valid voxels. Then, all valid voxels are traversed and filled into the corresponding positions of the empty sparse cost volume to obtain the sparse cost volume.

[0087] It should be noted that data structures can be used to optimize storage and access efficiency. For example, in one embodiment, a hash table or a sparse matrix can be used to optimize storage and access efficiency.

[0088] The pseudo-point generation method based on mixed light images provided in this embodiment first pairs infrared features, visible light features and lidar features in sequence based on voxel ratios to obtain multiple multimodal feature pairs, and then calculates the similarity of the multimodal feature pairs to generate valid voxels, and finally generates a sparse cost volume based on the coordinates of the valid voxels. In this way, by pairing features from different sensors and calculating similarities, valid voxels that are valuable for subsequent processing can be efficiently screened out, reducing redundant data and making information fusion more focused and efficient. Furthermore, only when features from different sensors show a high degree of consistency will the corresponding voxels be considered valid, so that erroneous or inconsistent data points can be eliminated, thereby improving the reliability of the overall data. In this way, by generating a sparse cost volume with high reliability and strong robustness through effective voxels, the reliability and accuracy of pseudo-point generation can be guaranteed.

[0089] S103 . Regularize the sparse cost volume, and generate target confidences and target offsets corresponding to the third depth map and the fourth depth map based on some sparse cost volumes in the multiple sparse cost volumes.

[0090] Specifically, a traditional regularization method can be used to process the sparse cost body, or a neural network regularization method can be used to process the sparse cost body, which is not limited in this embodiment. For example, in one embodiment, the sparse cost body can be regularized based on a convolutional neural network (CNN) or a graph convolutional network (GCN).

[0091] Furthermore, the regularized sparse cost volume contains the consistency information between different sensor data. By further processing this information, the target confidence and target offset can be generated.

[0092] Furthermore, target confidence is an indicator used to measure the reliability and accuracy of the depth map, and the target confidence can reflect the credibility of the data in the depth map. It should be noted that the data in the depth map with higher target confidence will be given a higher weight, while the data in the depth map with lower target confidence may be ignored or processed with a lower weight, thereby improving the quality of pseudo point generation.

[0093] Furthermore, the target offset is a vector value used to correct the error of the data in the depth map. By analyzing the consistency information in the sparse cost volume, the target offset of the data in the depth map can be calculated. It should be noted that the target offset indicates the direction and size of the adjustment of the data in the depth map. By applying the target offset, the estimation error caused by sensor noise, occlusion, illumination change and other factors can be corrected, thereby obtaining a more accurate pseudo point.

[0094] A specific embodiment is given below to introduce in detail the process of obtaining the target confidence and the target offset:

[0095] (1) Regularizing the sparse cost volume, extracting features of the sparse cost volume, and generating cost volume features.

[0096] For details, please continue to refer to Figure 2 , the sparse cost body is input into the adaptive depth correction module, and the convolution layer, pooling layer and other structures in the adaptive depth correction module are used to extract features from the regularized sparse cost body.

[0097] In the specific implementation, the adaptive depth correction module slides on the sparse cost volume to capture the spatial structure and texture information in the cost volume and generate the cost volume features.

[0098] (2) Based on the cost volume features, calculating the target confidence and the target offset of each pixel in the second depth map.

[0099] Specifically, the confidence of each pixel is calculated by the adaptive depth correction module using the cost volume features. In specific implementation, the adaptive depth correction module predicts the reliability score of the depth estimation of each pixel based on the information in the cost volume features and determines the corresponding target confidence.

[0100] Furthermore, the adaptive depth correction module calculates the offset that needs to be adjusted for each pixel based on the cost volume features. In specific implementation, the adaptive depth correction module predicts the direction and size of the depth estimation value of each pixel that needs to be adjusted based on the difference information in the cost volume features, and determines the corresponding target offset.

[0101] The pseudo point generation method based on the mixed light image provided in this embodiment first regularizes the sparse cost volume, extracts the features of the sparse cost volume, generates the cost volume features, and then calculates the target confidence and target offset of each pixel in the second depth map based on the cost volume features. In this way, by regularizing the sparse cost volume, the noise and irregularity in the cost volume can be effectively reduced, and the consistency and accuracy of the data can be improved. Further, the target confidence calculated based on the cost volume features can accurately reflect the reliability of the depth estimation value of each pixel in the third depth map and the fourth depth map, which helps to better utilize the depth information in subsequent processing. Further, the target offset calculated by the cost volume features can accurately guide the adjustment of the depth estimation value of each pixel in the second depth map. The direction and size of the offset are accurately calculated to ensure that the depth estimation value can be as close to the true value as possible. This precise offset guidance helps to improve the accuracy and robustness of the depth map. In this way, the performance of the entire depth map can be improved by the target confidence and target offset, and the reliability and accuracy of subsequent pseudo point generation can be guaranteed.

[0102] S104: Correct the third depth map and the fourth depth map based on the target confidence and the target offset to generate a target pseudo point.

[0103] Specifically, each pixel in the third depth map and the fourth depth map is weighted according to the target confidence. In specific implementation, the original depth value of the pixel with high target confidence can be maintained or enhanced, and the pixel with low target confidence can be weakened or ignored.

[0104] Furthermore, the target offset map is used to perform pixel-by-pixel offset correction on the weighted third depth map and the fourth depth map. In specific implementation, the depth value of each pixel is adjusted according to its corresponding offset to be closer to the actual depth. It should be noted that the direction and size of the target offset are calculated based on the features of the sparse cost volume, so it can accurately guide the depth correction.

[0105] Please continue to refer to Figure 2 , the third depth map and the fourth depth map output by the dual-spectrum depth completion module are input into the adaptive correction module, and after correction based on the target confidence and the target offset, they are output through the adaptive correction module to obtain the target pseudo point. Figure 3 This is a schematic diagram of a target pseudo point provided by an exemplary embodiment of the present application. Figure 2 and Figure 3 The infrared image and the first depth map, the visible light image and the second depth map are input into the model, and are processed by the dual-spectrum depth completion module, the sparse cost volume module and the adaptive depth completion module in sequence to output the corresponding target pseudo points.

[0106] A specific embodiment is given below to introduce the generation process of the target pseudo point in detail:

[0107] (1) Delete the pixel points whose target confidence is lower than a preset threshold.

[0108] Specifically, the specific value of the preset threshold is set according to actual needs, and is not limited in this embodiment. In specific implementation, each pixel in the third depth map and the fourth depth map is traversed to check whether its target confidence is lower than the preset threshold. If so, the pixel is deleted from the depth map or marked as invalid.

[0109] (2) Adjusting the pixel point based on the target offset.

[0110] In specific implementation, for each retained pixel, the direction and size of its depth value are adjusted according to its corresponding target offset.

[0111] (3) Back-projecting the corrected third depth map and the fourth depth map to obtain the target pseudo point.

[0112] In a specific implementation, the depth information in the two-dimensional image is mapped back to the three-dimensional space through back projection. In a specific implementation, a three-dimensional coordinate can be assigned to each pixel point to obtain a target pseudo point.

[0113] The pseudo-point generation method based on the mixed light image provided in this embodiment first deletes the pixels whose target confidence is lower than the preset threshold, then moves the pixels based on the target offset, and finally back-projects the corrected third depth map and the fourth depth map to obtain the target pseudo-point. In this way, by deleting the pixels whose target confidence is lower than the preset threshold, the outliers in the depth map caused by estimation errors or noise can be significantly reduced. Furthermore, by applying the calculated target offset to adjust the depth value of the pixel, the accuracy and reliability of the depth estimation can be further guaranteed. Furthermore, by deleting the pixels with low target confidence, the invalid calculation amount in the subsequent processing steps is reduced, so that the pseudo-point generation process can maintain a high calculation efficiency. In this way, the accuracy and reliability of pseudo-point generation are guaranteed by the target confidence and the target offset.

[0114] The pseudo point generation method based on mixed light image provided in this embodiment firstly deeply completes the first depth map and the second depth map, generates the third depth map and the fourth depth map, calculates the multimodal geometric points corresponding to the third depth map and the fourth depth map, then inputs the multimodal geometric points and the laser radar geometric points into the encoder, generates multiple sparse cost volumes, and then regularizes the sparse cost volumes, generates target confidence and target offset, and finally corrects the third depth map and the fourth depth map based on the target confidence and target offset to generate target pseudo points. In this way, by combining the multi-sensor fusion technology of visible light detectors, infrared detectors and laser radar detectors, high-quality pseudo points can be generated under adverse conditions, thereby enhancing the robustness of the system under different conditions. Further, the use of sparse operations significantly reduces the consumption of computing resources and improves the computing efficiency. By only performing feature extraction and cost volume calculation on multimodal geometric points and laser radar geometric points, the memory usage is reduced and the computing speed is improved. Further, the dual-spectrum depth completion module, the sparse cost volume module and the adaptive depth correction module are optimized for specific problems, and are independent of each other, which is convenient for subsequent maintenance and upgrading. In this way, the quality and efficiency of pseudo-point generation are significantly improved through the three-modality detectors, ensuring the reliability and accuracy of pseudo-points.

[0115] Corresponding to the aforementioned embodiment of a pseudo point generation method based on a mixed light image, the present application also provides an embodiment of a pseudo point generation device based on a mixed light image.

[0116] The embodiment of the pseudo-point generation device based on mixed light images of the present application can be applied to a pseudo-point generation device based on mixed light images. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of the pseudo-point generation device based on mixed light images reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram of a pseudo point generating device based on a mixed light image in the present application, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, the pseudo-point generating device based on the mixed light image in which the device in the embodiment is located may also include other hardware according to the actual function of the pseudo-point generating device based on the mixed light image, which will not be described in detail.

[0117] Figure 5 This is a structural diagram of the first embodiment of the pseudo point generation device based on mixed light image provided by this application. Figure 5 The device provided in this embodiment includes a processing module 510, a generating module 520 and a correcting module 530; wherein,

[0118] The processing module 510 is used to deeply complete the first depth map and the second depth map, generate a third depth map and a fourth depth map, and calculate multimodal geometric points corresponding to the third depth map and the fourth depth map;

[0119] The first depth map and the third depth map are infrared data, and the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map;

[0120] The generating module 520 is used to input the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes, each of which includes at least two spectral data, and the spectral data of each sparse cost volume has a different mixing ratio;

[0121] The generating module 520 is used to regularize the sparse cost volume, and generate the target confidence and target offset corresponding to the third depth map and the fourth depth map based on some sparse cost volumes in the multiple sparse cost volumes respectively;

[0122] The correction module 530 is used to correct the third depth map and the fourth depth map based on the target confidence and the target offset to generate a target pseudo point.

[0123] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0124] Optionally, the processing module 510 is specifically configured to input the infrared image and the first depth map into a dual-spectrum depth completion module to obtain multi-scale infrared features;

[0125] The processing module 510 is further specifically configured to input the visible light image and the second depth map into a dual-spectrum depth completion module to obtain multi-scale visible light features;

[0126] The processing module 510 is further specifically configured to generate the third depth map based on a majority of multi-scale infrared features, and to generate the fourth depth map based on a majority of multi-scale visible light features.

[0127] Optionally, the processing module 510 is further specifically configured to detect and identify the scene to be identified based on a laser radar, and generate a radar coordinate system;

[0128] The processing module 510 is further specifically configured to back-project the third depth map and the fourth depth map into the radar coordinate system to obtain the multimodal geometric point.

[0129] Optionally, the generating module 520 is specifically used to extract features from the multimodal geometric points and the laser radar geometric points to obtain target features; wherein the target features include infrared features, visible light features and laser radar features;

[0130] The generating module 520 is further specifically configured to obtain valid voxels and voxel ratios based on the target features, process coordinates corresponding to the valid voxels based on the voxel ratios, and generate a sparse cost volume.

[0131] Optionally, the generating module 520 is further specifically configured to sequentially pair the infrared feature, the visible light feature, and the laser radar feature based on the voxel ratio to obtain a plurality of multimodal feature pairs;

[0132] The generating module 520 is further specifically used to calculate the similarity of the multimodal feature pair and generate the valid voxel;

[0133] The generating module 520 is further specifically configured to generate the sparse cost volume based on the coordinates of the valid voxels.

[0134] Optionally, the generating module 520 is further specifically configured to correct the infrared feature based on the visible light feature and the laser radar feature to obtain the multimodal feature pair corresponding to the infrared feature; wherein, in the multimodal feature pair corresponding to the infrared feature, the infrared feature is the main body;

[0135] The generation module 520 is also specifically used to correct the visible light feature based on the infrared feature and the lidar feature to obtain the multimodal feature pair corresponding to the visible light feature; wherein, in the multimodal feature pair corresponding to the visible light feature, the visible light feature is the main body.

[0136] Optionally, the generating module 520 is further specifically used to regularize the sparse cost volume, extract features of the sparse cost volume, and generate cost volume features;

[0137] The generating module 520 is further specifically configured to calculate the target confidence and the target offset of each pixel in the second depth map based on the cost volume feature.

[0138] Optionally, the correction module 530 is specifically used to delete the pixel points whose target confidence is lower than a preset threshold;

[0139] The correction module 530 is specifically configured to adjust the pixel point based on the target offset;

[0140] The correction module 530 is further specifically used to back-project the corrected third depth map and the fourth depth map to obtain the target pseudo point.

[0141] Please continue to refer to Figure 4 The present application also provides a pseudo-point generation device based on a mixed light image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0142] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of any one of the methods provided in the present application are implemented.

[0143] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0144] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0145] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A pseudo point generation method based on mixed light image, characterized in that: The method comprises: Depth-complete the first depth map and the second depth map to generate a third depth map and a fourth depth map, and calculate multimodal geometric points corresponding to the third depth map and the fourth depth map; The first depth map and the third depth map are infrared data, and the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map; Inputting the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes, each of the sparse cost volumes comprising at least two spectral data, and the spectral data of each sparse cost volume having a different mixing ratio; Regularizing the sparse cost volume, and generating target confidences and target offsets corresponding to the third depth map and the fourth depth map based on some of the sparse cost volumes in the multiple sparse cost volumes; The third depth map and the fourth depth map are corrected based on the target confidence and the target offset to generate a target pseudo point, wherein the third depth map and the fourth depth map correspond to different partial sparse cost volumes.

2. The method according to claim 1, characterized in that The depth completion of the first depth map and the second depth map to generate a third depth map and a fourth depth map includes: Inputting the infrared image and the first depth map into a dual-spectrum depth completion module, and acquiring multi-scale infrared features of the infrared image based on the dual-spectrum depth completion module; Inputting the visible light image and the second depth map into a dual-spectrum depth completion module, and acquiring multi-scale visible light features of the visible light image based on the dual-spectrum depth completion module; The first depth map is completed based on the multi-scale infrared features to generate the third depth map, and the second depth map is completed based on the multi-scale visible light features to generate the fourth depth map.

3. The method according to claim 1, characterized in that The calculating the multimodal geometric points corresponding to the third depth map and the fourth depth map includes: Based on the laser radar detection and identification of the scene to be identified, a radar coordinate system is generated; The third depth map and the fourth depth map are back-projected into the radar coordinate system to obtain the multimodal geometric point.

4. The method according to claim 1, characterized in that: The step of inputting the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes includes: Performing feature extraction on the multimodal geometric points and the laser radar geometric points to obtain target features; wherein the target features include infrared features, visible light features and laser radar features; Valid voxels and voxel ratios are acquired based on the target features, coordinates corresponding to the valid voxels are processed based on the voxel ratios, and a sparse cost volume is generated.

5. The method according to claim 4, characterized in that The acquiring effective voxels and voxel ratios based on the target features, processing coordinates corresponding to the effective voxels based on the voxel ratios, and generating a sparse cost volume includes: Sequentially pairing the infrared feature, the visible light feature, and the laser radar feature based on the voxel ratio to obtain a plurality of multimodal feature pairs; Calculating the similarity of the multimodal feature pairs to generate the valid voxels; The sparse cost volume is generated based on the coordinates of the valid voxels.

6. The method according to claim 5, characterized in that The step of sequentially pairing the infrared feature, the visible light feature, and the laser radar feature based on the voxel ratio to obtain a plurality of multimodal feature pairs includes: Determining the spectrum type to be supplemented according to the spectrum type of the depth map to be corrected; Determine a sparse cost volume corresponding to the depth map to be corrected based on the spectrum type to be supplemented, wherein one sparse cost volume corresponds to one spectrum type to be supplemented; Based on all spectral types of the depth map to be corrected, the voxel ratio of each spectrum in each sparse cost volume corresponding to the depth map to be corrected is determined; wherein the sparse cost volume corresponding to each depth map to be corrected includes spectral information of all spectral types to be supplemented, and the spectral information of the maximum voxel ratio of the sparse cost volume corresponding to each depth map to be corrected is different.

7. The method according to claim 5, characterized in that The step of sequentially pairing the infrared feature, the visible light feature, and the laser radar feature based on the voxel ratio to obtain a plurality of multimodal feature pairs includes: The infrared feature is corrected based on the visible light feature and the laser radar feature to obtain the multimodal feature pair corresponding to the infrared feature; wherein the infrared feature is the main body in the multimodal feature pair corresponding to the infrared feature; The visible light feature is corrected based on the infrared feature and the laser radar feature to obtain the multimodal feature pair corresponding to the visible light feature; wherein, in the multimodal feature pair corresponding to the visible light feature, the visible light feature is the main body.

8. The method according to claim 1, characterized in that The regularizing the sparse cost volume to generate a target confidence and a target offset includes: Regularizing the sparse cost volume, extracting features of the sparse cost volume, and generating cost volume features; The target confidence and the target offset of each pixel in the second depth map are calculated based on the cost volume features.

9. The method according to claim 8, characterized in that The step of correcting the third depth map and the fourth depth map based on the target confidence and the target offset to generate a target pseudo point includes: Deleting the pixel points whose target confidence is lower than a preset threshold; Adjusting the pixel based on the target offset; The target pseudo point is obtained by back-projecting the corrected third depth map and the fourth depth map.

10. A pseudo point generation device based on mixed light image, characterized in that: The device comprises a processing module, a generating module and a correcting module; wherein, The processing module is used to deeply complete the first depth map and the second depth map, generate a third depth map and a fourth depth map, and calculate multimodal geometric points corresponding to the third depth map and the fourth depth map; The first depth map and the third depth map are infrared data, and the second depth map and the fourth depth map are visible light data; the density of the third depth map is greater than that of the first depth map, and the density of the fourth depth map is greater than that of the second depth map; The generating module is used to input the multimodal geometric points and the laser radar geometric points into an encoder to generate a plurality of sparse cost volumes; wherein each of the sparse cost volumes includes at least two spectral data, and the spectral data mixing ratios of the sparse cost volumes are different; The generating module is used to regularize the sparse cost volume, and generate the target confidence and target offset corresponding to the third depth map and the fourth depth map based on part of the sparse cost volumes in the multiple sparse cost volumes respectively; The correction module is used to correct the third depth map and the fourth depth map based on the target confidence and the target offset to generate a target pseudo point, wherein the third depth map and the fourth depth map correspond to different partial sparse cost volumes.

Images