Dense matching method and system based on superpixels and lidar geometric perception

Abstract

The application discloses a dense matching method and system based on superpixels and LiDAR geometric perception, and solves the problem of poor point cloud reconstruction effect in the prior art.The application projects point clouds to generate a sparse depth map, and divides an image into a plurality of superpixel regions; then, a plane is fitted for each superpixel region to form a stable geometric prior, thereby avoiding random initial errors and error accumulation; then, pixels are divided into reliable and unreliable categories according to the reliability of the fitted plane, and a differentiated propagation strategy is used for plane estimation; in this way, the classified propagation avoids over-smoothing of the depth map and large-area holes, and the superpixel plane geometric constraint is introduced into the joint cost, thereby enhancing the anti-interference ability under the conditions of light variation and weak texture; thus, the matching robustness and depth integrity under the conditions of weak texture and complex light in a large-scale urban scene are significantly improved, and the generated point cloud is more accurate, less missing and more detailed.

Description

Technical Field

[0001] This invention belongs to the field of photogrammetry and computer vision technology, specifically relating to a dense matching method and system based on superpixels and LiDAR geometric perception. Background Technology

[0002] Multi-view dense matching estimates depth maps and generates point clouds by calculating dense pixel correspondences on a set of image sequences with known poses; this method performs well in reconstructing textured regions. However, when faced with weakly textured regions and complex lighting variations commonly found in large-scale urban scenes, its core assumption of photometric consistency is easily compromised, leading to matching ambiguity and depth estimation errors, ultimately resulting in large-area missing areas and loss of detail in the reconstructed point cloud. In contrast, LiDAR (Light Detection and Ranging) can actively and directly acquire high-precision 3D point clouds of scenes, providing accurate geometric information for reconstruction tasks; however, LiDAR point clouds are often sparse and uneven in density, and the equipment is expensive and susceptible to occlusion, making it difficult to construct high-density, high-completeness models independently. Therefore, fusing the accurate geometric information provided by LiDAR with the rich textures contained in the images to achieve complementary advantages has become a key strategy to overcome the limitations of a single data source and generate high-quality, complete point clouds.

[0003] Currently, existing fusion methods can be divided into two categories: indirect fusion and direct fusion. Specifically, direct fusion methods for images and LiDAR point clouds are mainly divided into two types: image-assisted LiDAR interpolation and image-dense matching under LiDAR constraints. Among them, introducing LiDAR point clouds into the image-dense matching process can make up for the shortcomings of traditional image-assisted interpolation in predicting unstable regions with weak textures and large missing areas. The basic idea is to use sparse but high-precision LiDAR depth as prior information and embed it into the cost calculation or aggregation stage of the matching framework to guide the disparity or depth estimation to approach the true value during optimization. Specific methods include using LiDAR projection points to limit the local disparity search range to improve the reliability of the initial matching; or enhancing the matching cost corresponding to the true depth in the cost volume to assist subsequent optimization convergence.

[0004] Another approach uses multi-step propagation and local consistency constraints to diffuse LiDAR information to the surrounding area, thereby improving the matching stability and detail recovery capability of weak texture, shadow and repetitive structure regions under sparse point cloud conditions. However, existing methods are mostly based on binocular stereo frames, which are limited by the observation view and baseline configuration and have insufficient geometric redundancy. This means that even with the introduction of LiDAR constraints, matching uncertainty and incomplete depth may still occur in complex scenes, significant scale changes or severe occlusion.

[0005] In contrast, multi-view imagery, with its richer geometric constraints provided by multi-baseline observations, is more robust in reconstructing occlusion and complex structures. Recent research has begun to introduce sparse depth priors into multi-view reconstruction, fusing geometric and photometric information during cost construction or inference stages to improve stability under weak texture and complex imaging conditions. However, these methods mostly remain at the cost level, providing guidance or additional constraints. The influence of prior information in iterative propagation is limited, leading to a decrease in the distinguishability between different depth hypotheses when photometric consistency degrades. Erroneous solutions may still be retained and gradually spread, causing error accumulation and local geometric shifts, which in turn affect the point cloud reconstruction results. Therefore, how to achieve more sustainable and propagable geometric constraints during the matching optimization process remains the key to improving fusion accuracy and robustness. Summary of the Invention

[0006] The purpose of this invention is to provide a dense matching method and system based on superpixels and LiDAR geometric perception to solve the problem of poor point cloud reconstruction effect in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a dense matching method based on superpixels and LiDAR geometry sensing is provided, including: Acquire registered LiDAR sparse point cloud and image data; The LiDAR sparse point cloud is projected onto the image pixel plane corresponding to the image data to generate a sparse projection depth map. Superpixel segmentation is performed on the image data to obtain several superpixel regions; Based on the sparse projection depth map, a fitting plane corresponding to each superpixel region is generated, and the reliability of the fitting plane of each superpixel region is calculated. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels. The reliability of any fitting plane is used to characterize the degree to which the fitting plane approximates the depth of each pixel in the corresponding superpixel region. Iterative plane estimation is performed on reliable and unreliable planar pixels to obtain the actual planar parameters of each pixel in the image data, so as to generate an actual depth map for 3D reconstruction based on the actual planar parameters of each pixel. In any iteration, a plane estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to estimate the plane of unreliable planar pixels, and a joint cost estimation strategy based on superpixel plane geometric constraints is used to estimate the plane of reliable planar pixels.

[0008] Based on the above disclosure, this invention first projects the LiDAR sparse point cloud onto the image pixel plane to generate a sparse projection depth map. Then, it performs superpixel segmentation on the image data to generate superpixel regions with planar consistency. Next, it combines the sparse projection depth map to generate a fitting plane for each superpixel region. This initialization method ensures that each superpixel region possesses planar parameters conforming to its local geometry from the outset, effectively avoiding initial errors caused by random assignment and the accumulation of errors during subsequent propagation. This provides an accurate and stable geometric prior for the entire depth diffusion process (i.e., the introduced prior not only provides guidance or additional constraints at the cost level but also participates in the entire iterative plane estimation process). Then, based on the reliability of the fitting plane, each pixel in the superpixel region is divided into reliable planar pixels and unreliable planar pixels, and a differentiated propagation strategy is adopted. The method employs a plane estimation strategy. For reliable plane pixels, a joint cost estimation strategy based on superpixel plane geometric constraints is used for plane estimation. For unreliable plane pixels, a strategy based on fitting neighboring reliable pixels and a comprehensive cost estimation strategy is used for plane estimation. This method avoids excessive smoothing and large-area holes in the depth map by using classification propagation. At the same time, the introduction of superpixel plane geometric constraints into the joint cost gives the invention a certain degree of resistance to interference from illumination changes and texture scarcity. It can still maintain stable reconstruction performance under complex conditions such as urban scenes (i.e., it can still maintain stable reconstruction performance when photometric consistency degrades). Therefore, the invention significantly improves the matching robustness and depth integrity in weak texture regions and under complex illumination conditions in large-scale urban scenes. The generated point cloud has higher accuracy, lower missing rate, and richer details. Thus, it achieves deep complementarity between LiDAR geometric information and image texture advantages.

[0009] In one possible design, the LiDAR sparse point cloud is projected onto the image pixel plane corresponding to the image data to generate a sparse projection depth map, including: Obtain camera extrinsic and intrinsic parameters, and based on the camera extrinsic parameters, transform the 3D coordinates of each point cloud in the LiDAR sparse point cloud set to the reference camera coordinate system to obtain the 3D camera coordinates corresponding to each point cloud. The z-axis coordinate value in the 3D camera coordinates of any point cloud is taken as the depth of that point cloud. Based on the camera intrinsic parameters, the coordinates of each 3D camera are projected onto the image pixel plane to obtain the pixel coordinates of each point cloud. Obtain an initial depth map, wherein the size of the initial depth map is the same as the size of the image data, and the depth at all locations in the initial depth map is 0; For any point cloud, determine whether the pixel coordinates of that point cloud are valid coordinates; If so, determine whether the target coordinates correspond to multiple point clouds, where the target coordinates are the pixel coordinates of any one of the point clouds; If so, the minimum depth is selected from the depths of multiple point clouds, and the minimum depth is recorded at the position corresponding to the target coordinates in the initial depth map. Otherwise, the depth of any point cloud is recorded at the position corresponding to the target coordinates in the initial depth map, so that the sparse projection depth map is obtained after all point clouds have been traversed.

[0010] In one possible design, based on the sparse projected depth map, a fitting plane corresponding to each superpixel region is generated, including: For any superpixel region, the depth of each pixel in the superpixel region is determined based on the sparse projection depth map, and the pixels with a depth of non-zero are taken as the effective depth points in the superpixel region. Determine whether the number of valid depth points is greater than or equal to the number threshold; If so, then a local plane equation is fitted using the effective depth points, and the local plane equation is solved to obtain the plane parameters of the fitted plane corresponding to any superpixel region. Using planar parameters, a fitting plane corresponding to any superpixel region is constructed; Accordingly, the reliability of the fitting plane for each superpixel region is calculated. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels, including: For any superpixel region, the sum of squared fitting residuals is calculated based on the effective depth points within the superpixel region and the corresponding fitting plane, which serves as the reliability of the fitting plane corresponding to the superpixel region. Determine whether the reliability is less than or equal to a preset tolerance threshold; If so, the pixels in any superpixel region are classified as reliable planar pixels; otherwise, the pixels in any superpixel region are classified as unreliable planar pixels.

[0011] In one possible design, during any iteration, an estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to perform plane estimation for unreliable planar pixels, including: In the i-th iteration, for any unreliable planar pixel, a pixel-by-pixel spiral expansion search method with dynamic angle mutual exclusion is used to search for the set of reliable neighboring pixels of any unreliable planar pixel; Based on a reliable set of pixels in the neighborhood, a local neighborhood plane is fitted to obtain the plane parameters of the local neighborhood plane; Obtain the planar parameters of each reliable neighboring pixel in the reliable neighboring pixel set at the i-th iteration, where the initial value of i is 1, and when i is 1, the planar parameters of any reliable neighboring pixel at the i-th iteration are the planar parameters of the fitting plane corresponding to the superpixel region to which any reliable neighboring pixel belongs. The plane parameters of the local plane in the neighborhood and the plane parameters of each reliable pixel in the neighborhood at the i-th iteration are merged to generate a candidate plane parameter set; Calculate the matching cost of each group of candidate plane parameters in the candidate plane parameter set relative to any unreliable plane pixel, and take the minimum matching cost as the original cost; The candidate plane parameters corresponding to the original cost are used as the target parameters, and the target parameters are subjected to random perturbations of normal and depth to obtain the updated plane parameters. Calculate the matching cost of the updated plane parameters relative to any unreliable plane pixel to obtain the new matching cost; Determine if the cost of the new match is less than the original cost; If yes, then the updated planar parameters are used as the latest planar parameters for any unreliable planar pixel; otherwise, the target parameters are used as the latest planar parameters for any unreliable planar pixel. Determine if i has reached the maximum number of iterations; If not, the latest planar parameter is used as the planar parameter of any unreliable planar pixel in the (i+1)th iteration; otherwise, the latest planar parameter is used as the actual planar parameter of the pixel corresponding to any unreliable planar pixel.

[0012] In one possible design, a pixel-by-pixel spiral expansion search method with dynamic angular mutual exclusion is used to search for the set of reliable neighboring pixels of any unreliable planar pixel, including: Using any of the unreliable planar pixels as the center point, and employing a pixel-by-pixel spiral outward stepping method, the b-th scan is performed to obtain the current scanned pixel point; From the currently scanned pixels, select pixels that belong to reliable planar pixels as potential candidate points; Obtain the reliable pixel candidate set obtained from the (b-1)th scan; For any potential candidate point, a first direction vector is generated from the center point to the potential candidate point, and a second direction vector is generated from the center point to each candidate reliable pixel in the reliable pixel candidate set. Generate a first direction vector pointing from the center point to any of the potential candidate points; Calculate the cosine similarity between the first direction vector and each of the second direction vectors, and select the largest cosine similarity from all the cosine similarities. Calculate the average color difference of all pixels in the superpixel region to which any unreliable planar pixel belongs, and calculate the Euclidean distance between any potential candidate point and the center point in the color space; Determine whether the maximum cosine similarity is less than the cosine angle repulsion threshold, and whether the Euclidean distance is less than or equal to the average color difference; If so, add any potential candidate point to the reliable pixel candidate set obtained by the (b-1)th scan, and after traversing all potential candidate points, obtain the reliable pixel candidate set obtained by the bth scan. Determine if the search stopping condition is met; If not, increment b by 1 and perform the b-th scan again using a pixel-by-pixel spiral outward stepping method until the search stopping condition is met. Then, take the reliable pixel candidate set that meets the search stopping condition as the neighborhood reliable pixel set.

[0013] In one possible design, the matching cost of each set of candidate plane parameters in the candidate plane parameter set relative to any unreliable plane pixel is calculated, including: For any set of candidate plane parameters in the candidate plane parameter set, the matching cost of the any set of candidate plane parameters relative to any unreliable plane pixel is calculated according to the following formula; ; In the formula, This indicates that any set of candidate plane parameters is relative to any unreliable plane pixel. The matching cost, This represents the weight of any unreliable plane pixel. This indicates that any unreliable planar pixel in any set of candidate planar parameters The cost of normalized cross-correlation of multiple views under the following conditions Indicates the first The weight of each anchor pixel, Indicates the first Each anchor pixel in any set of candidate plane parameters The cost of normalized cross-correlation of multiple views under the following conditions The anchor pixel set is the set of points generated by back-projecting reliable pixels in each neighborhood onto the reference camera coordinate system. in, ,and For The center window is located in any set of candidate plane parameters Normalized cross-correlation values ​​of multiple views under induced homography transformation.

[0014] In one possible design, during any iteration, a joint cost estimation strategy based on superpixel planar geometric constraints is employed to perform planar estimation on reliable planar pixels, including: In the i-th iteration, for any reliable planar pixel, obtain the eight neighboring pixels of that reliable planar pixel, where the eight neighboring pixels are reliable planar pixels; Obtain the planar parameters of each eight-neighbor pixel in the i-th iteration, where the initial value of i is 1, and when i is 1, the planar parameters of any eight-neighbor pixel in the i-th iteration are the planar parameters of the fitting plane corresponding to the superpixel region to which any eight-neighbor pixel belongs. Using the planar parameters of each eight-neighbor pixel at the i-th iteration, a candidate planar parameter set is generated; Calculate the plane prior penalty cost for each set of candidate plane parameters in the candidate plane parameter set relative to any reliable plane pixel, wherein the plane prior penalty cost is used as a superpixel plane geometric constraint. Calculate the cost weight adjustment value for any reliable plane pixel; Based on the cost weight adjustment value of any reliable plane pixel and the prior penalty cost of each plane, the joint cost of each group of candidate plane parameters relative to any reliable plane pixel is calculated. The candidate plane parameter with the minimum joint cost is selected as the latest plane parameter for any reliable plane pixel; Determine if i has reached the maximum number of iterations; If not, the latest planar parameter is used as the planar parameter of any reliable planar pixel in the (i+1)th iteration; otherwise, the latest planar parameter is used as the actual planar parameter of the pixel corresponding to any reliable planar pixel.

[0015] In one possible design, the planar prior penalty cost for each set of candidate planar parameters in the candidate planar parameter set relative to any reliable planar pixel is calculated, including: For any set of candidate plane parameters, the plane prior penalty cost of the any set of candidate plane parameters relative to any reliable plane pixel is calculated according to the following formula; ; This represents the prior planar penalty cost of any set of candidate plane parameters relative to any reliable plane pixel. This represents the depth of any reliable plane pixel calculated using any set of candidate plane parameters. This represents the depth of any reliable planar pixel calculated using specified parameters, where the specified parameters are the planar parameters of the fitted plane corresponding to the superpixel region to which the reliable planar pixel belongs, and Indicates the scale factor; Accordingly, based on the cost weight adjustment value of any reliable plane pixel and the prior penalty cost of each plane, the joint cost of each group of candidate plane parameters relative to any reliable plane pixel is calculated, including: For any set of candidate plane parameters, the joint cost of the candidate plane parameters relative to any reliable plane pixel is calculated according to the following formula; ; This represents the joint cost of any set of candidate plane parameters relative to any reliable plane pixel. This represents the cost weight adjustment value for any reliable plane pixel. For any reliable planar pixel in any set of candidate planar parameters The cost of normalized cross-correlation of multiple views.

[0016] In one possible design, the cost weight adjustment value for any reliable planar pixel is calculated, including: Calculate the Euclidean space distance between any reliable planar pixel and the target point, wherein the target point is the center point of the superpixel region corresponding to any reliable planar pixel; The cost weight adjustment value is calculated using Euclidean spatial distance and a distance decay function based on a Gaussian kernel.

[0017] Secondly, a dense matching system based on superpixels and LiDAR geometry sensing is provided, including: A dense matching system based on superpixels and LiDAR geometry sensing, characterized in that it includes: The acquisition unit is used to acquire the registered LiDAR sparse point cloud and image data; The depth generation unit is used to project the LiDAR sparse point cloud onto the image pixel plane corresponding to the image data to generate a sparse projection depth map. The superpixel segmentation unit is used to perform superpixel segmentation on image data to obtain several superpixel regions; A pixel partitioning unit is used to generate a fitting plane corresponding to each superpixel region based on the sparse projection depth map, and to calculate the reliability of the fitting plane of each superpixel region. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels. The reliability of any fitting plane is used to characterize the degree to which the fitting plane approximates the depth of each pixel in the corresponding superpixel region. The plane estimation unit is used to perform iterative plane estimation on reliable and unreliable plane pixels to obtain the actual plane parameters of each pixel in the image data, so as to generate an actual depth map for 3D reconstruction based on the actual plane parameters of each pixel. In any iteration, an estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to estimate the plane of unreliable plane pixels, and a joint cost estimation strategy based on superpixel plane geometric constraints is used to estimate the plane of reliable plane pixels.

[0018] Thirdly, a dense matching device based on superpixels and LiDAR geometric perception is provided. Taking the device as an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the dense matching method based on superpixels and LiDAR geometric perception as described in the first aspect or any possible design of the first aspect.

[0019] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the dense matching method based on superpixels and LiDAR geometric perception as described in the first aspect or any possible design of the first aspect.

[0020] Fifthly, a computer program product containing instructions is provided that, when executed on a computer, causes the computer to perform the dense matching method based on superpixel and LiDAR geometric perception as described in the first aspect or any possible design of the first aspect.

[0021] Beneficial effects: (1) The present invention first projects LiDAR point cloud to generate sparse depth map and divides the image into several superpixel regions; then, fits a plane to each superpixel region to form a stable geometric prior, thereby avoiding random initial errors and error accumulation; then, according to the reliability of the fitted plane, the pixels are divided into reliable and unreliable categories, and a differential propagation strategy is adopted to perform plane estimation. In this way, classification propagation avoids excessive smoothing of the depth map and large-area holes. Moreover, the present invention introduces superpixel plane geometric constraints in the joint cost, thereby enhancing the anti-interference ability under illumination changes and weak texture conditions; thus, the present invention significantly improves the matching robustness and depth integrity under weak texture and complex illumination in large-scale urban scenes, thereby making the generated point cloud more accurate, less missing, and richer in detail. Therefore, it is very suitable for large-scale application and promotion. Attached Figure Description

[0022] Figure 1 A flowchart illustrating the dense matching method based on superpixels and LiDAR geometric perception provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the local search and allocation iteration of the superpixel center provided in an embodiment of the present invention; Figure 3 A schematic diagram of reliable pixel search provided for an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a dense matching system based on superpixels and LiDAR geometry sensing provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0024] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0025] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0026] Example: See Figure 1 As shown, the dense matching method based on superpixels and LiDAR geometric perception provided in the first aspect of this embodiment can be executed by a computer device with certain computing resources, but is not limited to. For example, the computer device can be, but is not limited to, a server, an edge computer, or a personal computer (PC, which refers to a multi-purpose computer of a size, price, and performance suitable for personal use; desktop computers, laptops, mini-laptops, tablets, and ultrabooks are all personal computers), a smartphone, or a personal digital assistant (PDA). It is understood that the aforementioned execution subject does not constitute a limitation on the embodiments of this application. Accordingly, the operation steps of this method can be, but are not limited to, the steps S1 to S5 below.

[0027] S1. Obtain the registered LiDAR sparse point cloud and image data; In this embodiment, for example, but not limited to, LiDAR can be used to obtain point cloud data of the target scene, thereby using the point cloud data obtained by LiDAR to form a LiDAR sparse point cloud; At the same time, multiple images (i.e., multi-view images) are acquired by a camera at the same time, and then one image is selected from the multiple images as the image data for dense matching.

[0028] Thus, after acquiring image data and LiDAR sparse point clouds, effective depth images can be generated based on the LiDAR sparse point clouds to provide a data foundation for the subsequent planar hypothesis; wherein, the process of generating depth images can be, but is not limited to, as shown in step S2 below.

[0029] S2. Project the LiDAR sparse point cloud onto the image pixel plane corresponding to the image data to generate a sparse projection depth map. In specific applications, examples include, but are not limited to, first projecting the point cloud onto the reference camera coordinate system, and then projecting it from the reference camera coordinate system onto the image pixel plane to generate a sparse projection depth map.

[0030] Optionally, the process of generating a depth map based on continuous coordinate transformation is as shown in steps S21 to S26 below.

[0031] S21. Obtain camera extrinsic and intrinsic parameters, and based on the camera extrinsic parameters, transform the three-dimensional coordinates of each point cloud in the LiDAR sparse point cloud set to the reference camera coordinate system to obtain the three-dimensional camera coordinates corresponding to each point cloud. The z-axis coordinate value in the three-dimensional camera coordinates of any point cloud is taken as the depth of that point cloud. In this embodiment, the camera extrinsic parameters may include, but are not limited to, rotation matrices and translation vectors, while the camera intrinsic parameters may include, but are not limited to, the focal length of the camera in the x and y directions (i.e., horizontal focal length and vertical focal length), as well as the x-coordinate and y-coordinate of the principal point.

[0032] Thus, after obtaining the aforementioned camera intrinsic and extrinsic parameters, the three-dimensional coordinates of each point cloud in the LiDAR sparse point cloud set can be transformed to the reference camera coordinate system based on the camera extrinsic parameters. For any point cloud, the three-dimensional coordinates of the point cloud can be transformed to the reference camera coordinate system according to, but is not limited to, the following formula.

[0033] ; In the formula, The rotation matrix is ​​a parameter in the camera's extrinsic parameters. Let be the point cloud coordinate vector of any point cloud. The translation vector in the camera's extrinsic parameters. Let be the camera coordinate vector of any point cloud, where Let T represent the 3D camera coordinates of any point cloud, where T represents the transpose operation. ,and This represents the three-dimensional coordinates of any point cloud.

[0034] Thus, based on the aforementioned formula, after transforming each point cloud to the reference camera coordinate system, the depth of each point cloud is obtained. Then, the three-dimensional camera coordinates of each point cloud can be projected onto the image pixel plane corresponding to the image data, thereby obtaining the position of each point cloud on the image pixel plane (i.e., pixel coordinates), so that depth recording can be performed based on the pixel coordinates to generate a sparse projection depth map. The secondary projection process is shown in step S22 below.

[0035] S22. Based on the camera intrinsic parameters, project each 3D camera coordinate onto the image pixel plane to obtain the pixel coordinates of each point cloud; in specific implementation, for any point cloud, for example, but not limited to, the 3D camera coordinates of the point cloud can be projected onto the image pixel plane according to the following formula.

[0036] ; In the formula, Let be the pixel coordinate vector of any point cloud. , This represents the pixel coordinates of any point cloud. , These represent the focal length of the camera in the x and y directions, respectively. , Let x and y be the principal coordinates of the camera. These represent the x-coordinate, y-coordinate, and z-axis coordinate in the 3D camera coordinate system of any point cloud.

[0037] Thus, based on the aforementioned formula, after obtaining the pixel coordinates of each point cloud in the image pixel plane, the depth can be assigned in the initial depth map according to the pixel coordinates, thereby obtaining a sparse projection depth map. The process is shown in steps S23 to S26 below.

[0038] S23. Obtain an initial depth map, wherein the size of the initial depth map is the same as the size of the image data, and the depth at all locations in the initial depth map is 0.

[0039] S24. For any point cloud, determine whether the pixel coordinates of the point cloud are valid coordinates. In specific implementation, if the pixel coordinates of any point cloud are not within the image range of the initial depth map, they are determined to be invalid coordinates; conversely, if they are within the image range of the initial depth map, they are determined to be valid coordinates. When the pixel coordinates of any point cloud are invalid coordinates, it means that there is no projection point at any position in the initial depth map. At this time, depth recording can be performed. At the same time, due to the discreteness of LiDAR point clouds and the influence of occlusion relationships, multiple point clouds may be projected onto the same pixel position. Therefore, when the pixel coordinates are valid coordinates, it is also necessary to determine whether the pixel coordinates correspond to multiple point clouds. The process is shown in step S25 below.

[0040] S25. If so, determine whether the target coordinates correspond to multiple point clouds, where the target coordinates are the pixel coordinates of any point cloud. In specific implementation, assume the pixel coordinates are (u1, v1). If point cloud A, point cloud B, and point cloud C are projected and their corresponding pixel coordinates are all (u1, v1), then it is determined that (u1, v1) corresponds to multiple point clouds. In this case, the minimum depth projection rule needs to be used to handle the conflict of the same pixel. The process is shown in step S26 below.

[0041] S26. If yes, select the minimum depth from the depths of multiple point clouds and record the minimum depth at the position corresponding to the target coordinates in the initial depth map; otherwise, record the depth of any point cloud at the position corresponding to the target coordinates in the initial depth map, so as to obtain the sparse projection depth map after all point clouds have been traversed.

[0042] In this embodiment, when multiple point clouds are projected onto the same pixel location, the minimum depth among the depths of the multiple point clouds is recorded as the depth of the same pixel location (i.e., initially, the depth at this location is 0, and then it is updated to the minimum depth). Conversely, if there is no conflict with the same pixel, the depth of any of the aforementioned point clouds can be directly recorded at its corresponding pixel coordinates. In this way, after traversing all point clouds in the aforementioned manner, a sparse projection depth map can be obtained. Of course, a location with a depth of 0 in the sparse projection depth map indicates that there is no projection point at that location.

[0043] Based on the aforementioned steps S21 to S26, after generating the sparse projection depth map, superpixel segmentation of the image data can be performed, as shown in step S3 below.

[0044] S3. Perform superpixel segmentation on the image data to obtain several superpixel regions. In specific implementation, after obtaining the sparse projection depth map, considering the requirement for region consistency in subsequent 3D plane fitting and to avoid computational expansion caused by pixel-by-pixel processing, this embodiment further utilizes the low-level visual features of the image, including color information and texture information, to perform superpixel segmentation. In this step, the original image (i.e., image data) is divided into several superpixel regions with prior plane consistency, and each superpixel region is used as the basic processing unit for subsequent operations.

[0045] In practical applications, firstly, a five-dimensional feature vector is constructed for each pixel in the image data; in practical applications, for any pixel, its corresponding five-dimensional feature vector is: ,in, This represents the luminance, green-red, and blue-yellow components of any pixel in the CIELAB color space (this color space is closer to the human eye's perception of color differences than the RGB color space), while This represents the x-coordinate and y-coordinate of any given pixel.

[0046] Secondly, define the joint distance between two pixels. This distance measures the overall similarity between two pixels in terms of color and space, and its calculation formula is as follows: ; in, This represents the color distance between pixel a and pixel c, defined as the Euclidean distance between them in the CIELAB 3D color space (the smaller the value, the closer the colors of the two pixels are). This represents the Euclidean distance between pixels a and c in the image pixel plane. This is a color normalization constant used to normalize the dimensions of color distances, eliminating the influence of differences in the value ranges of various components in the color space. This is a spatial normalization constant used to normalize the dimensions of spatial distances in order to balance the contributions of color distance and spatial distance to the joint distance.

[0047] Finally, to avoid and The complexity of the distance formula is ultimately expressed as: ; in, It represents the initial clustering step size. It is the total number of pixels in the image data. The preset number of superpixel targets; The compactness factor is a preset scalar weighting coefficient used to directly adjust the intensity ratio of color constraints and spatial constraints. Its preferred value range is 10 to 40.

[0048] Therefore, based on the aforementioned distance formula, the pixels in the image data can be clustered, and the process is as follows: Step 1: Determine the initial clustering step size, and evenly distribute several initial cluster centers on the plane corresponding to the image data according to the initial clustering step size. The horizontal and vertical spacing between adjacent initial cluster centers is equal to the initial clustering step size, and the formula for calculating the initial clustering step size is: .

[0049] Step 2: Correct the positions of each initial cluster center to obtain the corrected initial cluster centers.

[0050] In this embodiment, the process of correcting the initial cluster center is as follows: using the brightness component or grayscale image of the image data, the gradient magnitude of each pixel in the image data is calculated; then, for any initial cluster center, the pixel position with the smallest gradient magnitude is searched in its 3×3 neighborhood, and the cluster center is moved to that pixel position, thereby obtaining the corrected initial cluster center.

[0051] Step 3: Based on the corrected initial cluster centers, perform clustering processing on the pixels in the image data to obtain several initial clusters after the clustering process.

[0052] In specific implementation, the clustering process is as follows: (1) Obtain the cluster centers at the h-th iteration, where the initial value of h is 1, and when h is 1, the cluster centers at the h-th iteration are the corrected initial cluster centers; (2) For any cluster center, obtain the local neighborhood of any cluster center, where the local neighborhood of any cluster center is (2) Based on the five-dimensional feature vectors of each pixel in the local neighborhood, and using the aforementioned distance formula, calculate the distance between each pixel in the local neighborhood and any cluster center (where, Figure 2 Figure (a) shows a partial view. For the search process, please refer to Figure 2 (3) For any pixel in the local neighborhood, if the distance between the pixel and any cluster center is less than the minimum distance currently recorded for the pixel (which is the initial value in the first iteration, i.e., a preset value), then the pixel is assigned to the cluster corresponding to the cluster center, and the minimum distance of the pixel is updated to the distance between the pixel and the cluster center, so that after traversing all pixels in the local neighborhood, the division of each pixel in the local neighborhood is completed. In this way, after all cluster centers in the h-th iteration are traversed, each pixel is assigned to the nearest cluster center; (4) Determine whether the clustering stopping condition is met; (5) If not, calculate the arithmetic mean of the five-dimensional feature vectors of all pixels in the cluster corresponding to each cluster center, and use it as the new cluster center ( Figure 2 Figure (b) in the diagram illustrates the update of cluster centers. (See Figure [b] for details.) Figure 2 (b) (6) Take each new cluster center as the cluster center at the (h+1)th iteration; (7) Increment h by 1 and re-obtain the cluster center at the (h)th iteration until the clustering stopping condition is met, and obtain several initial clusters.

[0053] In this embodiment, the clustering stopping condition can be, but is not limited to, the change in the position of the cluster center being less than a preset threshold (i.e., the distance between each cluster center in two adjacent iterations is less than the preset threshold), or reaching the preset maximum number of clustering iterations (which can be set to 4-10 times).

[0054] After obtaining several initial clusters, cluster merging can be performed, and the process is as follows: Step 4: Collect clusters with fewer than a preset number of pixels from each initial cluster as isolated clusters and merge them into the initial clusters that are closest to the features of the isolated clusters. After merging, several superpixel clusters are obtained. In this embodiment, the initial cluster that is closest to the features can be the initial cluster that is closest to the cluster center of the isolated cluster.

[0055] In this way, after obtaining several superpixel clusters, the pixels in each superpixel cluster can be used to generate superpixel regions.

[0056] After obtaining the superpixel region, the aforementioned sparse projection depth map can be combined to perform plane hypothesis initialization and pixel classification, as shown in step S4 below.

[0057] S4. Based on the sparse projected depth map, generate a fitting plane corresponding to each superpixel region, and calculate the reliability of the fitting plane for each superpixel region. Based on the reliability of each fitting plane, classify the pixels in each superpixel region into reliable plane pixels and unreliable plane pixels. The reliability of any fitting plane is used to characterize the degree to which the fitting plane approximates the depth of each pixel in the corresponding superpixel region. In this embodiment, this step is equivalent to initializing a plane hypothesis for each superpixel region and classifying all pixels into two categories according to the reliability of the plane estimation: reliable plane pixels and unreliable plane pixels. This classification result will directly determine the role and processing method of each pixel in the subsequent depth propagation process. The fitting process of the plane corresponding to the superpixel region is shown in steps S41 to S44 below.

[0058] S41. For any superpixel region, based on the sparse projection depth map, determine the depth of each pixel in the superpixel region, and take the pixels with a depth of non-zero as the effective depth points in the superpixel region. In this embodiment, it has been explained above that the size of the sparse projection depth map is the same as the size of the image data (which is an image). Therefore, the depth at the corresponding pixel coordinate position can be determined in the sparse projection depth map according to the pixel coordinates of each pixel in the superpixel region, and thus be taken as the depth of each pixel. In this way, the pixels with a depth of non-zero can be taken as the effective depth points. Then, plane fitting can be performed based on the effective depth points, and the process is as follows: steps S42 to S44.

[0059] S42. Determine whether the number of valid depth points is greater than or equal to the number threshold. In specific implementation, the number threshold can be set to, but is not limited to, 3. When the number of valid depth points is less than 3, the pixels in any superpixel region can be directly regarded as unreliable plane pixels. Conversely, if the number of valid depth points is greater than or equal to 3, it is necessary to fit the local plane equation based on the valid depth points. The process is shown in step S43 below.

[0060] S43. If so, then using the effective depth points, fit a local plane equation, and solve the local plane equation to obtain the plane parameters of the fitted plane corresponding to any superpixel region.

[0061] Optionally, based on the depth and camera parameters corresponding to each effective depth point, the three-dimensional camera coordinates of each effective depth point are first back-projected into the reference camera coordinate system to obtain the three-dimensional camera coordinates of each effective depth point; then, based on the three-dimensional camera coordinates, the least squares method is used to fit the local plane equation.

[0062] Specifically, assuming the depth expression of a plane in the image coordinate system is: ; In the formula, Indicates depth, Let be the plane parameters to be determined. , This represents the x-coordinate and y-coordinate of each point in the 3D camera coordinate system on the plane; where the coordinates can be based on each effective depth point. Establish linear equations All effective depth points together form an overdetermined system of linear equations (i.e., local plane equations), which can be solved by minimizing the sum of squares of the depth errors: ; In the formula, This represents the planar parameters of the fitted plane corresponding to any superpixel region. Then, these represent the x and y coordinates in the 3D camera coordinates of the z-th effective depth point. This represents the depth of the z-th valid depth point (i.e., z represents the index of the valid depth point). It is a set of points with effective depth.

[0063] Thus, the least squares problem can be solved by constructing normal equations or singular value decomposition (SVD). Based on this, after the solution is completed, the plane parameters of the fitting plane corresponding to any of the aforementioned superpixel regions can be obtained. Then, based on this, the corresponding fitting plane can be constructed, as shown in step S44 below.

[0064] S44. Using planar parameters, construct the fitting plane corresponding to any superpixel region; in this embodiment, Substituting these values ​​into the aforementioned depth expression yields the corresponding fitted plane.

[0065] Thus, based on the aforementioned steps S41 to S44, after fitting the fitting plane corresponding to each superpixel region, the plane reliability can be estimated, as shown in the following steps S45 to S47.

[0066] S45. For any superpixel region, based on the effective depth points within that superpixel region and the corresponding fitting plane, calculate the sum of squared fitting residuals to determine the reliability of the fitting plane corresponding to that superpixel region. In specific implementation, the formula for calculating the sum of squared fitting residuals is: ; In the formula, This represents the sum of squared residuals of the fit for any superpixel region.

[0067] In this embodiment, the sum of squared residuals reflects the degree to which the fitted plane approximates the depth of each pixel within the corresponding superpixel region (i.e., the degree to which the effective depth point approximates the depth), which is the error between the fitted value and the true depth. The smaller the value, the more reliable the plane assumption; therefore, it can be used as the reliability of the fitted plane.

[0068] After calculating the reliability of the fitting plane corresponding to any superpixel region, the classification of each pixel in any superpixel region can be performed based on this, as shown in steps S46 and S47 below.

[0069] S46. Determine whether the reliability is less than or equal to a preset tolerance threshold; In this embodiment, when the reliability is less than or equal to the preset tolerance threshold, the fitted plane is marked as a reliable plane, and all pixels inside it are classified as reliable plane pixels; otherwise, it is marked as an unreliable plane, and all pixels inside it are classified as unreliable plane pixels; wherein, the classification process is as shown in step S47 below.

[0070] S47. If yes, then the pixels in any superpixel region are classified as reliable planar pixels; otherwise, the pixels in any superpixel region are classified as unreliable planar pixels.

[0071] Thus, for any superpixel region, the number of effective depth points must be greater than or equal to a threshold, and the reliability of the corresponding fitted plane must be less than or equal to a preset tolerance threshold, in order for the pixels within it to be classified as reliable planar pixels.

[0072] Specifically, for any pixel Let its superpixel region be... Then its category label Defined as: ; In the formula, A set consisting of valid depth points, meaning the number of valid depth points in the set must be greater than or equal to a certain threshold. , This indicates the preset tolerance threshold.

[0073] Based on the aforementioned steps S41 to S47, after dividing the pixels in each superpixel region into reliable planar pixels and unreliable planar pixels, different depth propagation strategies can be adopted for different categories of pixels, as shown in step S5 below.

[0074] S5. Iterative plane estimation is performed on reliable and unreliable planar pixels to obtain the actual planar parameters of each pixel in the image data, so as to generate an actual depth map for 3D reconstruction based on the actual planar parameters of each pixel. In any iteration, a plane estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to estimate the plane of unreliable planar pixels, and a joint cost estimation strategy based on superpixel plane geometric constraints is used to estimate the plane of reliable planar pixels. In this embodiment, after pixel classification is completed, in each iteration, a corresponding depth estimation strategy is adopted according to the different categories to which the pixels belong.

[0075] For unreliable planar pixels, the accurate geometric information carried by reliable planar pixels needs to be propagated to surrounding unreliable planar pixels, thereby achieving incremental diffusion from sparse LiDAR priors to dense depth maps. Specifically, firstly, reliable pixels are searched for using a pixel-by-pixel spiral expansion with dynamic angular mutual exclusion within the spatial neighborhood. Then, the planar parameters of each reliable pixel are back-projected into 3D space sequentially, and the planar parameters are re-estimated from the reliable pixels using the least squares method, and the comprehensive cost is calculated. Finally, the latest planar parameters for this iteration are determined based on the comprehensive cost. In this way, unreliable pixels can achieve robust assignment through fitting with reliable pixels in the neighborhood and evaluation of the comprehensive cost.

[0076] Optionally, in any iteration, when the selected pixel is an unreliable planar pixel, an estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is adopted. The process of performing planar estimation on the unreliable planar pixel can be, but is not limited to, the steps shown below.

[0077] S51. In the i-th iteration, for any unreliable planar pixel, a pixel-by-pixel spiral expansion search method with dynamic angle mutual exclusion is used to search for the set of reliable neighboring pixels of any unreliable planar pixel.

[0078] In specific applications, examples can be taken, but are not limited to, using the steps S51a to S51j shown below.

[0079] S51a. Using any of the unreliable planar pixels as the center point, perform the b-th scan using a pixel-by-pixel spiral outward stepping method to obtain the current scanned pixel point; in specific implementation, the pixel-by-pixel spiral outward stepping method is as follows: starting from the aforementioned center point, expand outward sequentially according to a spiral path, visiting other pixels in the neighborhood pixel by pixel, and the search radius of each step increases sequentially until the pixels that meet the conditions are covered or the set maximum search radius is reached.

[0080] See Figure 3As shown, the step-by-pixel spiral outward stepping method is as follows: starting from the upper right endpoint of the pixel box where the center point is located in the image data, the search proceeds along the upper right diagonal direction with a step size of b pixels (initially b is 1) to obtain a scan point (i.e. Figure 3 pixels in ); then, from the pixel Among pixels in the same row, select the pixel that is one pixel away from the center point along the top-left object line and use it as the first endpoint. Similarly, start from the pixel... From the pixels in the same column, select pixels that are one pixel away from the center point along the lower right object line, and designate them as the second endpoint; then, from the pixels in the same column as the first endpoint, select pixels that are one pixel away from the center point along the lower left object line, and designate them as the third endpoint; finally, select the pixels... With the first endpoint, pixel The pixels between the second endpoint, the first endpoint and the third endpoint, and the second endpoint and the third endpoint are taken as the currently scanned pixels. Figure 3 medium pixel The pixels between the current dashed frame and the innermost dashed frame are used as the currently scanned pixels.

[0081] After obtaining the current scanned pixel point obtained from the b-th scan, potential candidate points can be selected from the current scanned pixel points, as shown in step S51b below.

[0082] S51b. From the currently scanned pixels, select pixels that belong to reliable planar pixels as potential candidate points.

[0083] After filtering out potential candidate points, a reliable set of pixel candidates obtained from the previous scan can be obtained, as shown in step S51c below.

[0084] S51c. Obtain the reliable pixel candidate set obtained from the (b-1)th scan; In this embodiment, when b is 1, the reliable pixel candidate set obtained from the (b-1)th scan is an empty set. The reason for introducing the reliable pixel candidate set obtained from the (b-1)th scan is to force the collected reliable pixels to have spatial orientation balance and avoid geometric orientation deviation. That is, it is necessary to introduce dynamic angle mutual exclusion test for each potential candidate point. The process is as shown in the following steps S51d to S51h.

[0085] S51d. For any potential candidate point, generate a first direction vector pointing from the center point to the potential candidate point, and generate a second direction vector pointing from the center point to each candidate reliable pixel in the reliable pixel candidate set. In this embodiment, the coordinates of the potential candidate point, the coordinates of the center point, and the coordinates of each candidate reliable pixel are known. Therefore, the first direction vector pointing from the center point to the potential candidate point and the second direction vector pointing from the center point to each candidate reliable pixel can be generated based on the aforementioned coordinates. Then, the cosine similarity between the first direction vector and each second direction vector can be calculated based on the aforementioned first direction vector and each second direction vector, as shown in step S51e below.

[0086] S51e. Calculate the cosine similarity between the first direction vector and each of the second direction vectors, and select the largest cosine similarity from among the cosine similarities; after calculating the cosine similarity between the first direction vector and each of the second direction vectors, the largest cosine similarity can be selected; then, it is also necessary to calculate the Euclidean distance between any potential candidate point and the center pixel in the CIELAB color space, the process of which is shown in step S51f below.

[0087] S51f. Calculate the average color difference of all pixels in the superpixel region to which any unreliable planar pixel belongs, and calculate the Euclidean distance between any potential candidate point and the center point in the color space. In specific implementation, first calculate the mean vector of the color vectors (CIELAB color vectors) of all pixels in the superpixel region to which any unreliable planar pixel belongs, and then take the magnitude of the mean vector as the average color difference. Then, based on the color vector of any potential candidate point and the color vector of the center point, calculate the Euclidean distance between them in the CIELAB color space. Finally, the maximum cosine similarity can be combined to determine whether any potential candidate point is a new candidate reliable pixel point. The process is shown in step S51g below.

[0088] S51g. Determine whether the maximum cosine similarity is less than the cosine angle repulsion threshold, and whether the Euclidean distance is less than or equal to the average color difference; in this embodiment, when the maximum cosine similarity is less than the cosine angle repulsion threshold (e.g., set to...), the maximum cosine similarity is determined by... If the Euclidean distance is less than or equal to the average color difference, then any potential candidate point can be added to the reliable pixel candidate set obtained by the aforementioned (b-1)th scan; otherwise, if the aforementioned conditions are not met, any potential candidate point will not enter the reliable pixel candidate set; in this way, after all potential candidate points have been traversed in the aforementioned manner, the reliable pixel candidate set obtained by the current scan can be obtained, and the process is shown in step S51h below.

[0089] S51h. If so, add any potential candidate point to the reliable pixel candidate set obtained by the (b-1)th scan, and obtain the reliable pixel candidate set obtained by the bth scan after traversing all potential candidate points.

[0090] After completing the b-th scan, it can be determined whether the search stopping condition is met, as shown in step S51i below.

[0091] S51i. Determine whether the search stopping condition is met; in this embodiment, the search stopping condition may be, but is not limited to, the number of candidate reliable pixels in the reliable pixel candidate set obtained by the b-th scan reaching a preset number (e.g., set to 8), or the spiral search radius reaching the maximum window radius (e.g., 100 pixels, where the radius is the value of b).

[0092] If the aforementioned search stopping condition is not met, a next scan is required, that is, the step size needs to be increased, and then the aforementioned steps S51a to S51i are re-executed until the search stopping condition is met, then a reliable set of neighborhood pixels can be obtained. The continuous search process is shown in step S51j below.

[0093] S51j. If not, increment b by 1 and perform the b-th scan again using a pixel-by-pixel spiral outward stepping method until the search stopping condition is met. Then, take the reliable pixel candidate set that meets the search stopping condition as the neighborhood reliable pixel set.

[0094] In this embodiment, when b is 2, the pixel-by-pixel spiral outward stepping method is as follows: See Figure 3 As shown, starting from the upper left endpoint of the pixel frame containing the center point in the image data, a scan point is obtained by searching along the upper left diagonal with a step size of b pixels (where b becomes 2). Figure 3 pixels in ,Right now (Two pixels apart from the center point); then, from the pixel point Among pixels in the same row, select pixels that are 2 pixels apart from the center point along the upper right object line and use them as the first endpoint. Similarly, select pixels from the same row... From the pixels in the same column, select pixels that are 2 pixels apart from the center point along the lower left object line, and designate them as the second endpoint; then, from the pixels in the same column as the first endpoint, select pixels that are 2 pixels apart from the center point along the lower right object line, and designate them as the third endpoint; finally, select the pixels... With the first endpoint, pixel The pixels between the second endpoint, the first endpoint and the third endpoint, and the second endpoint and the third endpoint are taken as the currently scanned pixels. Figure 3 medium pixel The dashed box containing the pixel The pixels between the dashed boxes are used as the current scanning pixels. In this way, the second scan is completed and a second set of reliable pixel candidates is obtained. If the set is not satisfactory, the step size needs to be increased and the scan continues.

[0095] The process of the third pixel-by-pixel spiral outward step is as follows: See Figure 3 As shown, starting from the lower left endpoint of the pixel frame containing the center point in the image data, and with a step size of b pixels (where b becomes 3), the search proceeds along the lower left diagonal to obtain a scan point (i.e., Figure 3 pixels in ,Right now (Three pixels apart from the center point); then, from the pixel point Among pixels in the same row, select pixels that are 3 pixels apart from the center point along the bottom right object line and use them as the first endpoint. Similarly, select pixels from the same row... From the pixels in the same column, select pixels that are 3 pixels apart from the center point along the top-left object line, and use them as the second endpoint; then, from the pixels in the same column as the first endpoint, select pixels that are 3 pixels apart from the center point along the top-right object line, and use them as the third endpoint; finally, select the pixels... With the first endpoint, pixel The pixels between the second endpoint, the first endpoint and the third endpoint, and the second endpoint and the third endpoint are taken as the currently scanned pixels. Figure 3 medium pixel The dashed box containing the pixel The pixels between the dashed boxes are taken as the current scanning pixels. In this way, the third scan is completed and a reliable set of pixels for the third scan is obtained. If the set is not satisfactory, the step size needs to be increased and the scan continues.

[0096] When b is 4, the search starts from the lower right corner of the pixel frame containing the center point in the image data, and proceeds along the lower right diagonal with a step size of 3 pixels to obtain a scan point. Figure 3 (not shown in the image), and then, following the same method described above, three endpoints are determined to determine the current scan pixel. After the current scan is completed, a spiral step is completed. In the next scan, the search starts from the upper right endpoint of the pixel box where the center point is located in the image data and searches along the upper right diagonal until the search stopping condition is met, then a reliable set of neighborhood pixels can be obtained.

[0097] Based on the aforementioned steps S51a to S51j, after determining the set of reliable neighboring pixels for any unreliable plane pixel, a local neighborhood plane can be fitted based on this, as shown in step S52 below.

[0098] S52. Based on the reliable neighboring pixel set, fit a local neighborhood plane to obtain the plane parameters of the local neighborhood plane. In specific implementation, firstly, back-project each reliable neighboring pixel in the reliable neighboring pixel set to the reference camera coordinates to obtain several anchor point pixels. Then, based on the several anchor point pixels, fit the local neighborhood plane to determine the plane parameters corresponding to the local neighborhood plane. Of course, the fitting process can be referred to the aforementioned step S43, and will not be repeated here.

[0099] After obtaining the planar parameters of the local plane in the neighborhood, the planar parameters of each reliable pixel in the neighborhood at the i-th iteration can be obtained, as shown in step S53 below.

[0100] S53. Obtain the planar parameters of each reliable neighboring pixel in the reliable neighboring pixel set at the i-th iteration, where the initial value of i is 1, and when i is 1, the planar parameter of any reliable neighboring pixel at the i-th iteration is the planar parameter of the fitting plane corresponding to the superpixel region to which the reliable neighboring pixel belongs; in this embodiment, when performing pixel classification as described above, if any fitting plane is classified as a reliable plane, then the pixel in the superpixel region corresponding to the fitting plane is a reliable plane pixel. Therefore, each reliable plane pixel corresponds to a reliable fitting plane. Thus, when i is 1, the planar parameter of each reliable neighboring pixel is the planar parameter of the fitting plane corresponding to its respective superpixel region.

[0101] After obtaining the planar parameters of each reliable neighboring pixel at the i-th iteration, they can be merged with the planar parameters of the local neighboring plane to obtain a candidate planar parameter set, as shown in step S54 below.

[0102] S54. Merge the planar parameters of the local plane in the neighborhood and the planar parameters of each reliable pixel in the neighborhood at the i-th iteration to generate a candidate planar parameter set.

[0103] After obtaining the candidate plane parameter set, the matching cost (i.e., the comprehensive cost) of each group of candidate plane parameters in the candidate plane parameter set relative to any unreliable plane pixel can be calculated. Based on the matching cost, the plane parameter of any unreliable plane pixel after the i-th iteration can be determined. The process is shown in step S55 below.

[0104] S55. Calculate the matching cost of each group of candidate plane parameters in the candidate plane parameter set relative to any unreliable plane pixel, and take the minimum matching cost as the original cost.

[0105] In specific implementation, for any set of candidate plane parameters in the candidate plane parameter set, the matching cost of the any set of candidate plane parameters relative to any unreliable plane pixel is calculated according to the following formula; ; In the formula, This indicates that any set of candidate plane parameters is relative to any unreliable plane pixel. The matching cost, This represents the weight of any unreliable plane pixel. This indicates that any unreliable planar pixel in any set of candidate planar parameters The cost of normalized cross-correlation of multiple views under the following conditions Indicates the first The weight of each anchor pixel (i.e.) (Indicates the pixel number of the anchor point). Indicates the first Each anchor pixel in any set of candidate plane parameters The cost of normalized cross-correlation of multiple views under the following conditions The anchor pixel set is the set of points generated by back-projecting each reliable neighboring pixel onto the reference camera coordinate system (i.e., the set of points obtained by back-projecting each reliable neighboring pixel onto the reference camera coordinate system).

[0106] in, ,and For The window centered on any set of candidate plane parameters Normalized cross-correlation values ​​of multiple views under induced homography transformation.

[0107] In practice, The calculation process is as follows: As explained above, when acquiring image data, the camera captures multiple images simultaneously. Therefore, when calculating... The window centered on any set of candidate plane parameters When normalizing the cross-correlation value of multiple views under induced homography transformation, it is necessary to first select images with overlapping areas with the image data from multiple images, and then use the selected images to form a view set; then, select the U-th image from multiple images as the reference view, with the initial value of U being 1.

[0108] Next, based on the reference view and the view set, and using the following formula, the value is calculated. The window centered on any set of candidate plane parameters Normalized cross-correlation values ​​of multiple views under induced homography transformation.

[0109] ; In the formula, Represents a collection of views. For view indexing, For any of the unreliable planar pixels The center window (can be set to be the center) (a 5x5 local window centered on the center) For the nth view in the view collection, Indicates a reference view. This represents the combination of any set of candidate plane parameters. The induced homography transformation matrix from the reference view to the nth view. Represents the window in the reference view pixels within pixel values, Represents a window in the target image pixels within The pixel values, where the target image is the nth view. The image obtained after homography transformation, and Represents the window in the reference view Average pixel value within, Represents the window in the nth view. The average pixel value within.

[0110] After calculating the multi-view normalized cross-correlation value with the U-th image as the reference view, U is incremented by 1, and the U-th image is reselected as the reference view until U equals W, resulting in W multi-view normalized cross-correlation values. Finally, the mean of the W multi-view normalized cross-correlation values ​​is used as the reference value. The window centered on any set of candidate plane parameters The normalized cross-correlation value of multiple views under induced homography transformation, where W is the total number of images acquired by the camera at the same time.

[0111] In this embodiment, The calculation process and The calculation process is the same, and will not be repeated here.

[0112] Furthermore, the formula for calculating the weight of any unreliable plane pixel is as follows: ; In the formula, For any of the unreliable planar pixels The central window, It is the pixel index within this window. For any unreliable planar pixel RGB color feature vector in the reference view For pixels RGB color feature vector in the reference view For spatial bandwidth (set to 2.5 in this embodiment), The color bandwidth (in this embodiment, it is set according to the pixel value range of the reference view, and is set to 35% of the pixel value range).

[0113] Furthermore, the first The weight of each anchor pixel depends on its relationship with any unreliable plane pixel. The color difference in the reference view is intended to prevent edge misinterpretation caused by crossing depth boundaries. The specific calculation formula is as follows: ; In the formula, Indicates the first The RGB color feature vector of each anchor pixel in the reference view A constant parameter to control color sensitivity, This represents the L2 norm.

[0114] Thus, based on the aforementioned formula, after calculating the matching cost of each group of candidate plane parameters relative to any unreliable plane pixel, the minimum matching cost can be selected as the original cost. Then, in order to further optimize the plane parameters, it is necessary to apply perturbation to the candidate plane parameters corresponding to the original cost, so as to recalculate the matching cost of the updated plane parameters. The process is shown in steps S56 and S57 below.

[0115] S56. The candidate plane parameters corresponding to the original cost are used as the target parameters, and the normal and depth of the target parameters are randomly perturbed to obtain the updated plane parameters. In this embodiment, the original plane is generated based on the candidate plane parameters corresponding to the original cost. Then, a random angle rotation is applied to the normal vector of the original plane (a small angle can be selected for perturbation, such as a random value between [0°, 10°]), and zero-mean Gaussian noise is added to the depth of each point in the original plane to form the updated plane, thereby obtaining the updated plane parameters.

[0116] After obtaining the updated planar parameters, the matching cost relative to any unreliable planar pixel can be calculated, as shown in step S57 below.

[0117] S57. Calculate the matching cost of the updated plane parameters relative to any unreliable plane pixel to obtain the new matching cost; in this embodiment, the calculation process of the matching cost can be referred to the aforementioned step S55, and the process will not be repeated here.

[0118] After obtaining the new matching cost, it can be compared with the original cost, and then the parameter corresponding to the minimum cost is selected as the latest plane parameter of any unreliable plane pixel. The process is shown in steps S58 and S59 below.

[0119] S58. Determine whether the cost of the new match is less than the original cost.

[0120] S59. If yes, then the updated planar parameters are used as the latest planar parameters for any unreliable planar pixel; otherwise, the target parameters are used as the latest planar parameters for any unreliable planar pixel.

[0121] Thus, after the plane estimation of each unreliable planar pixel in the i-th iteration is completed based on the aforementioned method, it can be determined whether the maximum number of iterations has been reached, as shown in step S510 below.

[0122] S510. Determine whether i has reached the maximum number of iterations; in this embodiment, the maximum number of iterations is 4; of course, during an iteration, the determination to stop the iteration is only made after all unreliable plane pixels and all reliable plane pixels have completed plane estimation.

[0123] When i has not reached the maximum number of iterations, the latest plane parameter is used as the plane parameter in the next iteration, and the aforementioned steps S51 to S510 are re-executed until the maximum number of iterations is reached. Then, the actual plane parameter of the pixel corresponding to any unreliable plane pixel can be obtained. The process is shown in step S511 below.

[0124] S511. If not, then the latest planar parameter is used as the planar parameter of any unreliable planar pixel in the (i+1)th iteration; otherwise, the latest planar parameter is used as the actual planar parameter of the pixel corresponding to any unreliable planar pixel.

[0125] Thus, through the aforementioned steps S51 to S511, the plane parameters of each unreliable plane pixel can be estimated in one iteration.

[0126] Furthermore, when the selected pixel is a reliable planar pixel in one iteration, a joint cost estimation strategy based on superpixel planar geometric constraints is adopted to estimate the plane of the reliable planar pixel, as shown in steps S512 to S520 below.

[0127] S512. In the i-th iteration, for any reliable plane pixel, obtain the eight neighboring pixels of that reliable plane pixel, where the eight neighboring pixels are reliable plane pixels.

[0128] After obtaining reliable planar pixels in the eight-neighborhood, the planar parameters of the eight-neighborhood pixels at the i-th iteration can be obtained, as shown in step S513 below.

[0129] S513. Obtain the planar parameters of each eight-neighbor pixel in the i-th iteration, where the initial value of i is 1, and when i is 1, the planar parameters of any eight-neighbor pixel in the i-th iteration are the planar parameters of the fitting plane corresponding to the superpixel region to which any eight-neighbor pixel belongs.

[0130] After obtaining the planar parameters of each eight-neighbor pixel at the i-th iteration, candidate planar parameters can be generated based on these parameters, as shown in step S514 below.

[0131] S514. Using the planar parameters of each eight-neighbor pixel at the i-th iteration, generate a candidate planar parameter set.

[0132] After generating candidate plane parameters, this embodiment introduces superpixel plane geometric constraints, that is, calculates the plane prior penalty cost to constrain the current plane hypothesis to be consistent with the initial plane parameters of its superpixel. The calculation process is shown in step S515 below.

[0133] S515. Calculate the planar prior penalty cost for each set of candidate planar parameters in the candidate planar parameter set relative to any reliable planar pixel, wherein the planar prior penalty cost is used as a superpixel planar geometric constraint.

[0134] In practical applications, for any set of candidate plane parameters, the prior plane penalty cost of the set of candidate plane parameters relative to any reliable plane pixel can be calculated using, for example, but not limited to, the following formula.

[0135] ; This represents the prior planar penalty cost of any set of candidate plane parameters relative to any reliable plane pixel. This represents the depth of any reliable plane pixel calculated using any set of candidate plane parameters (i.e., taking any set of candidate plane parameters as the plane parameters to be determined in the aforementioned depth expression, and then substituting the coordinates of any reliable plane pixel in the reference camera coordinates into the depth expression to obtain the corresponding depth). This represents the depth of any reliable planar pixel calculated using specified parameters, wherein the specified parameters... Let be the planar parameters of the fitted plane corresponding to the superpixel region to which any reliable planar pixel belongs, and Represents the scale factor.

[0136] in, ,and The distance between two adjacent assumed depths is a preset constant.

[0137] Thus, based on the aforementioned formula, after calculating the prior penalty cost of each group of candidate plane parameters relative to any reliable plane pixel, the cost weight adjustment value of any reliable plane pixel can be calculated, as shown in step S516 below.

[0138] S516. Calculate the cost weight adjustment value for any reliable planar pixel; in specific implementation, the cost weight adjustment value is used to adjust the photometric consistency matching cost and the aforementioned planar prior penalty cost. The calculation process is as follows: first, calculate the Euclidean space distance between any reliable planar pixel and the target point, where the target point is the center point of the superpixel region corresponding to any reliable planar pixel; then, using the Euclidean space distance and employing a distance decay function based on a Gaussian kernel, calculate the cost weight adjustment value.

[0139] Optionally, for example, the formula for calculating the cost weight adjustment value is: ; In the formula, This represents the cost weight adjustment value. and These represent the lower bound and upper bound of the geometric prior weights, respectively (with values ​​of 0 and 1). This represents the Euclidean spatial distance. The bandwidth parameter (in this embodiment, it is set to the average equivalent radius of the superpixel region corresponding to any reliable planar pixel) is used. , (This refers to the total number of pixels in the superpixel region corresponding to any reliable planar pixel).

[0140] After calculating the cost weight adjustment value, the joint cost of each set of candidate plane parameters relative to any reliable plane pixel can be calculated by combining the aforementioned prior penalty costs of each plane, as shown in step S517 below.

[0141] S517. Based on the cost weight adjustment value of any reliable plane pixel and the prior penalty cost of each plane, calculate the joint cost of each group of candidate plane parameters relative to any reliable plane pixel.

[0142] In practice, for any set of candidate plane parameters, the joint cost of the candidate plane parameters relative to any reliable plane pixel can be calculated using, for example, but not limited to, the following formula.

[0143] ; This represents the joint cost of any set of candidate plane parameters relative to any reliable plane pixel. This represents the cost weight adjustment value for any reliable plane pixel. For any reliable planar pixel in any set of candidate planar parameters The cost of normalized cross-correlation of multiple views.

[0144] In this embodiment, The calculation process can be found in step S55 above, and will not be repeated here.

[0145] After calculating the joint cost of each group of candidate plane parameters relative to any reliable plane pixel, the candidate plane parameter corresponding to the smallest joint cost can be selected as the latest plane parameter of any reliable plane pixel, as shown in step S518 below.

[0146] S518. Select the candidate plane parameter with the minimum joint cost as the latest plane parameter of any reliable plane pixel; after obtaining the latest plane parameter of any reliable plane pixel, it can be determined whether the iteration can be terminated, and the process is as shown in step S519 below.

[0147] S519. Determine whether i has reached the maximum number of iterations; In this embodiment, if i has not reached the number of iterations, the latest plane parameter of any reliable plane pixel is used as its parameter in the next iteration, and then the plane estimation is performed again until the maximum number of iterations is reached; otherwise, it is used as the actual plane parameter, and the process is as shown in step S520 below.

[0148] S520. If not, the latest planar parameter is used as the planar parameter of any reliable planar pixel in the (i+1)th iteration; otherwise, the latest planar parameter is used as the actual planar parameter of the pixel corresponding to any reliable planar pixel.

[0149] In this embodiment, as previously explained, the decision to stop the iteration is made only after all unreliable planar pixels and all reliable planar pixels have completed planar estimation. Therefore, when all unreliable and reliable planar pixels have completed planar estimation in one iteration and i has not reached the maximum number of iterations, the latest planar pixels corresponding to each unreliable and reliable planar pixel can be used as their parameters in the next iteration. Then, the estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost, or the joint cost estimation strategy based on superpixel planar geometric constraints, is re-adopted to perform planar estimation until the maximum number of iterations is reached, thus obtaining the actual planar parameters of each pixel.

[0150] Then, based on the actual planar parameters of each pixel, the actual depth of each pixel can be determined, thus obtaining the actual depth map; finally, the 3D reconstruction of the target scene can be performed based on the actual depth map.

[0151] Therefore, through the dense matching method based on superpixels and LiDAR geometric perception described in detail in steps S1 to S5 above, the present invention has the following beneficial effects: (1) The present invention divides the influencing data into superpixel regions with planar consistency, and then combines sparse LiDAR projection depth maps, and assigns a clear planar hypothesis to each superpixel region through least squares fitting; thus, this initialization method ensures that each superpixel has planar parameters that conform to the local geometry from the beginning, effectively avoiding the initial error caused by random assignment and the accumulation of errors in subsequent propagation, and providing an accurate and stable geometric prior for the entire depth diffusion process.

[0152] (2) Classification propagation is used to avoid excessive smoothing and large-area holes in the depth map. This invention divides pixels into reliable and unreliable categories and adopts different propagation strategies for each category: reliable pixels are estimated by a joint cost estimation strategy based on superpixel planar geometric constraints, which ensures the global smoothness of the planar assumption; while unreliable pixels are estimated by fitting and comprehensive cost evaluation strategies of neighboring reliable pixels, which can achieve robust assignment.

[0153] (3) It exhibits strong robustness to the sparsity of LiDAR point clouds and image quality. Unlike traditional methods that heavily rely on dense LiDAR point clouds or high-quality images, this invention only requires sparse LiDAR projection points as priors, and can fill large-scale deep holes through superpixel plane priors and iterative propagation mechanisms. At the same time, the superpixel plane geometric constraints introduced in the joint cost enable this method to have a certain degree of resistance to interference from lighting changes and texture scarcity, and can still maintain stable reconstruction performance under complex conditions such as urban scenes.

[0154] like Figure 4 As shown, the second aspect of this embodiment provides a software system for implementing the dense matching method based on superpixels and LiDAR geometric perception described in the first aspect of the embodiment, comprising: The acquisition unit is used to acquire the registered LiDAR sparse point cloud and image data.

[0155] The depth generation unit is used to project the LiDAR sparse point cloud onto the image pixel plane corresponding to the image data to generate a sparse projection depth map.

[0156] The superpixel segmentation unit is used to perform superpixel segmentation on image data to obtain several superpixel regions.

[0157] A pixel partitioning unit is used to generate a fitting plane corresponding to each superpixel region based on the sparse projection depth map, and to calculate the reliability of the fitting plane of each superpixel region. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels. The reliability of any fitting plane is used to characterize the degree to which the fitting plane approximates the depth of each pixel in the corresponding superpixel region.

[0158] The plane estimation unit is used to perform iterative plane estimation on reliable and unreliable plane pixels to obtain the actual plane parameters of each pixel in the image data, so as to generate an actual depth map for 3D reconstruction based on the actual plane parameters of each pixel. In any iteration, an estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to estimate the plane of unreliable plane pixels, and a joint cost estimation strategy based on superpixel plane geometric constraints is used to estimate the plane of reliable plane pixels.

[0159] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0160] like Figure 5 As shown, the third aspect of this embodiment provides a dense matching device based on superpixels and LiDAR geometric perception. Taking the device as an electronic device as an example, it includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the dense matching method based on superpixels and LiDAR geometric perception as described in the first aspect of the embodiment.

[0161] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0162] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee transceiver (a low-power LAN protocol based on the IEEE 802.15.4 standard), a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0163] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0164] The fourth aspect of this embodiment provides a storage medium that stores instructions containing the dense matching method based on superpixels and LiDAR geometric perception as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the dense matching method based on superpixels and LiDAR geometric perception as described in the first aspect of the embodiment.

[0165] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0166] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0167] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the dense matching method based on superpixel and LiDAR geometric perception as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0168] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dense matching method based on superpixels and LiDAR geometry sensing, characterized in that, include: Acquire registered LiDAR sparse point cloud and image data; The LiDAR sparse point cloud is projected onto the image pixel plane corresponding to the image data to generate a sparse projection depth map. Superpixel segmentation is performed on the image data to obtain several superpixel regions; Based on the sparse projection depth map, a fitting plane corresponding to each superpixel region is generated, and the reliability of the fitting plane of each superpixel region is calculated. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels. The reliability of any fitting plane is used to characterize the degree to which the fitting plane approximates the depth of each pixel in the corresponding superpixel region. Iterative plane estimation is performed on reliable and unreliable plane pixels to obtain the actual plane parameters of each pixel in the image data, so as to generate an actual depth map for 3D reconstruction based on the actual plane parameters of each pixel. In any iteration, a plane estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to estimate the plane of unreliable plane pixels, and a joint cost estimation strategy based on superpixel plane geometric constraints is used to estimate the plane of reliable plane pixels. In any iteration, an estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to perform plane estimation for unreliable plane pixels, including: In the i-th iteration, for any unreliable planar pixel, a pixel-by-pixel spiral expansion search method with dynamic angle mutual exclusion is used to search for the set of reliable neighboring pixels of any unreliable planar pixel; Based on a reliable set of pixels in the neighborhood, a local neighborhood plane is fitted to obtain the plane parameters of the local neighborhood plane; Obtain the planar parameters of each reliable neighboring pixel in the reliable neighboring pixel set at the i-th iteration, where the initial value of i is 1, and when i is 1, the planar parameters of any reliable neighboring pixel at the i-th iteration are the planar parameters of the fitting plane corresponding to the superpixel region to which any reliable neighboring pixel belongs. The plane parameters of the local plane in the neighborhood and the plane parameters of each reliable pixel in the neighborhood at the i-th iteration are merged to generate a candidate plane parameter set; Calculate the matching cost of each group of candidate plane parameters in the candidate plane parameter set relative to any unreliable plane pixel, and take the minimum matching cost as the original cost; The candidate plane parameters corresponding to the original cost are used as the target parameters, and the target parameters are subjected to random perturbations of normal and depth to obtain the updated plane parameters. Calculate the matching cost of the updated plane parameters relative to any unreliable plane pixel to obtain the new matching cost; Determine if the cost of the new match is less than the original cost; If yes, then the updated planar parameters are used as the latest planar parameters for any unreliable planar pixel; otherwise, the target parameters are used as the latest planar parameters for any unreliable planar pixel. Determine if i has reached the maximum number of iterations; If not, the latest planar parameter is used as the planar parameter of any unreliable planar pixel in the (i+1)th iteration; otherwise, the latest planar parameter is used as the actual planar parameter of the pixel corresponding to any unreliable planar pixel.

2. The dense matching method based on superpixels and LiDAR geometric perception according to claim 1, characterized in that, The LiDAR sparse point cloud is projected onto the image pixel plane corresponding to the image data to generate a sparse projection depth map, including: Obtain camera extrinsic and intrinsic parameters, and based on the camera extrinsic parameters, transform the 3D coordinates of each point cloud in the LiDAR sparse point cloud set to the reference camera coordinate system to obtain the 3D camera coordinates corresponding to each point cloud. The z-axis coordinate value in the 3D camera coordinates of any point cloud is taken as the depth of that point cloud. Based on the camera intrinsic parameters, the coordinates of each 3D camera are projected onto the image pixel plane to obtain the pixel coordinates of each point cloud. Obtain an initial depth map, wherein the size of the initial depth map is the same as the size of the image data, and the depth at all locations in the initial depth map is 0; For any point cloud, determine whether the pixel coordinates of that point cloud are valid coordinates; If so, determine whether the target coordinates correspond to multiple point clouds, where the target coordinates are the pixel coordinates of any one of the point clouds; If so, the minimum depth is selected from the depths of multiple point clouds, and the minimum depth is recorded at the position corresponding to the target coordinates in the initial depth map. Otherwise, the depth of any point cloud is recorded at the position corresponding to the target coordinates in the initial depth map, so that the sparse projection depth map is obtained after all point clouds have been traversed.

3. The dense matching method based on superpixels and LiDAR geometric perception according to claim 1, characterized in that, Based on the sparse projected depth map, a fitting plane corresponding to each superpixel region is generated, including: For any superpixel region, the depth of each pixel in the superpixel region is determined based on the sparse projection depth map, and the pixels with a depth of non-zero are taken as the effective depth points in the superpixel region. Determine whether the number of valid depth points is greater than or equal to the number threshold; If so, then a local plane equation is fitted using the effective depth points, and the local plane equation is solved to obtain the plane parameters of the fitted plane corresponding to any superpixel region. Using planar parameters, a fitting plane corresponding to any superpixel region is constructed; Accordingly, the reliability of the fitting plane for each superpixel region is calculated. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels, including: For any superpixel region, the sum of squared fitting residuals is calculated based on the effective depth points within the superpixel region and the corresponding fitting plane, which serves as the reliability of the fitting plane corresponding to the superpixel region. Determine whether the reliability is less than or equal to a preset tolerance threshold; If so, the pixels in any superpixel region are classified as reliable planar pixels; otherwise, the pixels in any superpixel region are classified as unreliable planar pixels.

4. The dense matching method based on superpixels and LiDAR geometric perception according to claim 1, characterized in that, A pixel-by-pixel spiral expansion search method with dynamic angle mutual exclusion is used to search for the reliable pixel set in the neighborhood of any unreliable planar pixel, including: Using any of the unreliable planar pixels as the center point, and employing a pixel-by-pixel spiral outward stepping method, the b-th scan is performed to obtain the current scanned pixel point; From the currently scanned pixels, select pixels that belong to reliable planar pixels as potential candidate points; Obtain the reliable pixel candidate set obtained from the (b-1)th scan; For any potential candidate point, a first direction vector is generated from the center point to the potential candidate point, and a second direction vector is generated from the center point to each candidate reliable pixel in the reliable pixel candidate set. Calculate the cosine similarity between the first direction vector and each of the second direction vectors, and select the largest cosine similarity from all the cosine similarities. Calculate the average color difference of all pixels in the superpixel region to which any unreliable planar pixel belongs, and calculate the Euclidean distance between any potential candidate point and the center point in the color space; Determine whether the maximum cosine similarity is less than the cosine angle repulsion threshold, and whether the Euclidean distance is less than or equal to the average color difference; If so, add any potential candidate point to the reliable pixel candidate set obtained by the (b-1)th scan, and after traversing all potential candidate points, obtain the reliable pixel candidate set obtained by the bth scan. Determine if the search stopping condition is met; If not, increment b by 1 and perform the b-th scan again using a pixel-by-pixel spiral outward stepping method until the search stopping condition is met. Then, take the reliable pixel candidate set that meets the search stopping condition as the neighborhood reliable pixel set.

5. The dense matching method based on superpixels and LiDAR geometric perception according to claim 1, characterized in that, Calculate the matching cost of each group of candidate plane parameters in the candidate plane parameter set relative to any unreliable plane pixel, including: For any set of candidate plane parameters in the candidate plane parameter set, the matching cost of the any set of candidate plane parameters relative to any unreliable plane pixel is calculated according to the following formula; ； In the formula, This indicates that any set of candidate plane parameters is relative to any unreliable plane pixel. The matching cost, This represents the weight of any unreliable plane pixel. This indicates that any unreliable planar pixel in any set of candidate planar parameters The cost of normalized cross-correlation of multiple views under the following conditions Indicates the first The weight of each anchor pixel, Indicates the first Each anchor pixel is in any set of candidate plane parameters The cost of normalized cross-correlation of multiple views under the following conditions The anchor pixel set is the set of points generated by back-projecting reliable pixels in each neighborhood onto the reference camera coordinate system. in, ,and For The center window is located in any set of candidate plane parameters Normalized cross-correlation values ​​of multiple views under induced homography transformation.

6. The dense matching method based on superpixels and LiDAR geometric perception according to claim 1, characterized in that, In any iteration, a joint cost estimation strategy based on superpixel planar geometric constraints is employed to perform planar estimation on reliable planar pixels, including: In the i-th iteration, for any reliable planar pixel, obtain the eight neighboring pixels of that reliable planar pixel, where the eight neighboring pixels are reliable planar pixels; Obtain the planar parameters of each eight-neighbor pixel in the i-th iteration, where the initial value of i is 1, and when i is 1, the planar parameters of any eight-neighbor pixel in the i-th iteration are the planar parameters of the fitting plane corresponding to the superpixel region to which any eight-neighbor pixel belongs. Using the planar parameters of each eight-neighbor pixel at the i-th iteration, a candidate planar parameter set is generated; Calculate the plane prior penalty cost for each set of candidate plane parameters in the candidate plane parameter set relative to any reliable plane pixel, wherein the plane prior penalty cost is used as a superpixel plane geometric constraint. Calculate the cost weight adjustment value for any reliable plane pixel; Based on the cost weight adjustment value of any reliable plane pixel and the prior penalty cost of each plane, the joint cost of each group of candidate plane parameters relative to any reliable plane pixel is calculated. The candidate plane parameter with the minimum joint cost is selected as the latest plane parameter for any reliable plane pixel; Determine if i has reached the maximum number of iterations; If not, the latest planar parameter is used as the planar parameter of any reliable planar pixel in the (i+1)th iteration; otherwise, the latest planar parameter is used as the actual planar parameter of the pixel corresponding to any reliable planar pixel.

7. The dense matching method based on superpixels and LiDAR geometric perception according to claim 6, characterized in that, Calculate the prior planar penalty cost for each set of candidate plane parameters in the candidate plane parameter set relative to any reliable plane pixel, including: For any set of candidate plane parameters, the plane prior penalty cost of the any set of candidate plane parameters relative to any reliable plane pixel is calculated according to the following formula; ； This represents the prior planar penalty cost of any set of candidate plane parameters relative to any reliable plane pixel. This represents the depth of any reliable plane pixel calculated using any set of candidate plane parameters. This represents the depth of any reliable planar pixel calculated using specified parameters, where the specified parameters are the planar parameters of the fitted plane corresponding to the superpixel region to which the reliable planar pixel belongs, and Indicates the scale factor; Accordingly, based on the cost weight adjustment value of any reliable plane pixel and the prior penalty cost of each plane, the joint cost of each group of candidate plane parameters relative to any reliable plane pixel is calculated, including: For any set of candidate plane parameters, the joint cost of the candidate plane parameters relative to any reliable plane pixel is calculated according to the following formula; ； This represents the joint cost of any set of candidate plane parameters relative to any reliable plane pixel. This represents the cost weight adjustment value for any reliable plane pixel. For any reliable planar pixel in any set of candidate planar parameters The cost of normalized cross-correlation of multiple views.

8. The dense matching method based on superpixels and LiDAR geometric perception according to claim 6, characterized in that, Calculating the cost weight adjustment value for any reliable plane pixel includes: Calculate the Euclidean space distance between any reliable planar pixel and the target point, wherein the target point is the center point of the superpixel region corresponding to any reliable planar pixel; The cost weight adjustment value is calculated using Euclidean spatial distance and a distance decay function based on a Gaussian kernel.

9. A dense matching system based on superpixels and LiDAR geometry sensing, characterized in that, An apparatus for performing the dense matching method based on superpixel and LiDAR geometry sensing as described in any one of claims 1 to 8, wherein the apparatus comprises: The acquisition unit is used to acquire the registered LiDAR sparse point cloud and image data; The depth generation unit is used to project the LiDAR sparse point cloud onto the image pixel plane corresponding to the image data to generate a sparse projection depth map. The superpixel segmentation unit is used to perform superpixel segmentation on image data to obtain several superpixel regions; A pixel partitioning unit is used to generate a fitting plane corresponding to each superpixel region based on the sparse projection depth map, and to calculate the reliability of the fitting plane of each superpixel region. Based on the reliability of each fitting plane, the pixels in each superpixel region are divided into reliable plane pixels and unreliable plane pixels. The reliability of any fitting plane is used to characterize the degree to which the fitting plane approximates the depth of each pixel in the corresponding superpixel region. The plane estimation unit is used to perform iterative plane estimation on reliable and unreliable plane pixels to obtain the actual plane parameters of each pixel in the image data, so as to generate an actual depth map for 3D reconstruction based on the actual plane parameters of each pixel. In any iteration, an estimation strategy based on neighborhood reliable pixel fitting and comprehensive cost is used to estimate the plane of unreliable plane pixels, and a joint cost estimation strategy based on superpixel plane geometric constraints is used to estimate the plane of reliable plane pixels.

Images