Image processing method and device

Abstract

The invention relates to an image processing method and device, and the method comprises the steps: projecting point cloud data to an image coordinate system corresponding to an RGB image, and generating a sparse laser radar depth map which is aligned with the pixels of the RGB image; inputting the RGB image into a pre-trained monocular depth estimation network to obtain an output initial monocular depth map; performing global linear transformation on the initial monocular depth map based on laser radar depth measurement values on all effective pixel positions in the sparse laser radar depth map to obtain an intermediate depth map; in an image area corresponding to effective pixels identified by the sparse laser radar depth map, performing optimization calculation on the depth value of the middle depth map to generate a local optimization depth map; and generating a final fusion depth map based on the intermediate depth map and the local optimization depth map, thereby solving the problems of insufficient reliability of an RGB-D camera, large sparse blind area of a solid-state laser radar and fuzzy monocular depth scale in the prior art.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology

[0002] Environmental depth perception is the core foundation for robots to achieve autonomous navigation and interaction. Currently, it mainly relies on two types of sensor technologies: RGB-D cameras and solid-state LiDAR. However, each of these technologies has inherent limitations that are difficult to overcome.

[0003] Among them, although RGB-D cameras can directly provide pixel-level dense depth maps, their effective range is short, and their performance drops sharply in strong outdoor light or when facing transparent or highly reflective surfaces, severely limiting their reliability. Although solid-state lidar has accurate ranging and strong anti-interference capabilities, its scanning mode results in extremely sparse point cloud data, and its vertical field of view is usually narrow, making it difficult to cover the entire scene, especially for distant and low-reflectivity objects, where there are large-scale perception blind spots. Summary of the Invention

[0004] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an image processing method and apparatus, which solves the technical problems of insufficient reliability of RGB-D cameras and large sparse blind zones of solid-state LiDAR in the prior art.

[0005] (II) Technical Solution To achieve the above objectives, the main technical solutions adopted by the present invention include: In a first aspect, embodiments of the present invention provide an image processing method, comprising: Simultaneously acquire RGB images captured by an RGB-D camera and point cloud data captured by a solid-state LiDAR; The point cloud data is projected onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image. In the sparse LiDAR depth map, the pixel positions that are successfully projected are marked as valid pixels, and the valid pixels store the corresponding LiDAR depth measurement values. The pixel positions that are not successfully projected are marked as invalid pixels. The RGB image is input into a pre-trained monocular depth estimation network to obtain the initial monocular depth map output. A global linear transformation is performed on the initial monocular depth map based on the lidar depth measurement values ​​at all effective pixel locations in the sparse lidar depth map to obtain an intermediate depth map. Within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, the depth values ​​of the intermediate depth map are optimized to generate a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, in the optimization calculation, the depth value of the locally optimized depth map at its same coordinate position approaches the LiDAR depth measurement value stored in that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth values ​​of the intermediate depth map in the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in the texture information. The final fused depth map is generated based on the intermediate depth map and the locally optimized depth map.

[0006] In one possible embodiment, the monocular depth estimation network is a convolutional neural network with an encoder-decoder structure or a visual Transformer network.

[0007] In one possible embodiment, an intermediate depth map is obtained by performing a global linear transformation on the initial monocular depth map based on the lidar depth measurements at all valid pixel locations in the sparse lidar depth map. This includes: obtaining depth estimates of the initial monocular depth map at the same locations in the sparse lidar depth map at all valid pixel locations; determining a set of linear transformation parameters based on the depth estimates and the corresponding lidar depth measurements; and performing a global linear transformation on the initial monocular depth map using the linear transformation parameters to obtain the intermediate depth map.

[0008] In one possible embodiment, generating a final fused depth map based on an intermediate depth map and a locally optimized depth map includes: generating the final fused depth map according to the markings of the sparse LiDAR depth map; wherein, for each location marked as an invalid pixel, its depth value is taken from the corresponding location in the intermediate depth map; and for each location marked as a valid pixel, its depth value is taken from the corresponding location in the locally optimized depth map.

[0009] In one possible embodiment, the image processing method further includes: performing reliability verification on the fused depth map; wherein the reliability verification includes at least one of the following: consistency verification based on consecutive frames and sensor motion information, consistency verification based on lidar measurement data, and verification based on the continuity of the depth map itself.

[0010] In a second aspect, embodiments of the present invention provide an image processing apparatus, comprising: The acquisition module is used to simultaneously acquire RGB images captured by an RGB-D camera and point cloud data captured by a solid-state LiDAR. The first generation module is used to project point cloud data onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image. In the sparse LiDAR depth map, the pixel positions that are successfully projected are marked as valid pixels, and the valid pixels store the corresponding LiDAR depth measurement values. The pixel positions that are not successfully projected are marked as invalid pixels. The input module is used to input RGB images into a pre-trained monocular depth estimation network to obtain an initial monocular depth map. The global linear transformation module is used to perform a global linear transformation on the initial monocular depth map based on the lidar depth measurement values ​​at all effective pixel positions in the sparse lidar depth map to obtain an intermediate depth map. The optimization calculation module is used to optimize the depth values ​​of the intermediate depth map within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, generating a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, in the optimization calculation, the depth value of the locally optimized depth map at its same coordinate position approaches the LiDAR depth measurement value stored in that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth values ​​of the intermediate depth map in the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in the texture information. The second generation module is used to generate the final fused depth map based on the intermediate depth map and the locally optimized depth map.

[0011] In one possible embodiment, the monocular depth estimation network is a convolutional neural network with an encoder-decoder structure or a visual Transformer network.

[0012] In one possible embodiment, the global linear transformation module is configured to: obtain depth estimates of the initial monocular depth map at the same locations at all valid pixel locations of the sparse LiDAR depth map; determine a set of linear transformation parameters based on the depth estimates and the corresponding LiDAR depth measurements; and perform a global linear transformation on the initial monocular depth map using the linear transformation parameters to obtain an intermediate depth map.

[0013] In one possible embodiment, the second generation module is specifically configured to: generate a final fused depth map based on the markings of the sparse lidar depth map; wherein, for each location marked as an invalid pixel, its depth value is taken from the corresponding location in the intermediate depth map; and for each location marked as a valid pixel, its depth value is taken from the corresponding location in the locally optimized depth map.

[0014] In one possible embodiment, the image processing apparatus further includes: The reliability verification module is used to verify the reliability of the fused depth map; wherein, the reliability verification includes at least one of the following: consistency verification based on continuous frames and sensor motion information, consistency verification based on lidar measurement data, and verification based on the continuity of the depth map itself.

[0015] (III) Beneficial Effects The beneficial effects of this invention are: This application provides an image processing method and apparatus. It synchronously acquires RGB images captured by an RGB-D camera and point cloud data acquired by a solid-state LiDAR. The point cloud data is projected onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image. In the sparse LiDAR depth map, successfully projected pixel positions are marked as valid pixels, and each valid pixel stores a corresponding LiDAR depth measurement value. Pixel positions that are not successfully projected are marked as invalid pixels. The RGB image is input into a pre-trained monocular depth estimation network to obtain an initial output monocular depth map. A global linear transformation is performed on the initial monocular depth map based on the LiDAR depth measurement values ​​at all valid pixel positions in the sparse LiDAR depth map to obtain an intermediate depth map. Within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, the depth values ​​of the intermediate depth map are optimized to generate a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: For each effective pixel in the sparse LiDAR depth map, the optimization calculation aims to make the depth value of the locally optimized depth map at its same coordinate position approach the LiDAR depth measurement value stored in that effective pixel; Based on the texture information of the RGB image, a smoothing constraint is applied to the depth value of the intermediate depth map during the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in texture information. Finally, based on the intermediate depth map and the locally optimized depth map, a final fused depth map is generated, thereby solving the problems of insufficient reliability of RGB-D cameras, large sparse blind zones of solid-state LiDAR, and blurred monocular depth scale in existing technologies.

[0016] To make the above-mentioned objects, features and advantages to be achieved by the embodiments of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments of this disclosure will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of an image processing method provided in an embodiment of this application is shown; Figure 2 A structural block diagram of an image processing apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0019] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Currently, in addition to RGB-D cameras and solid-state LiDAR, existing technologies also include monocular depth estimation based on deep learning. This technology can predict scene depth from a single RGB image, breaking the distance limitations of physical sensors. However, its predictions suffer from a fundamental scale ambiguity problem; it can only recover relative depth relationships and cannot determine the absolute physical distance to any point. This makes it unsuitable for tasks requiring precise geometric information.

[0021] Based on this, embodiments of this application provide an image processing method and apparatus. This method involves simultaneously acquiring RGB images captured by an RGB-D camera and point cloud data acquired by a solid-state LiDAR, and projecting the point cloud data onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image. In the sparse LiDAR depth map, successfully projected pixel positions are marked as valid pixels, and each valid pixel stores a corresponding LiDAR depth measurement value. Pixel positions that are not successfully projected are marked as invalid pixels. The method also involves inputting the RGB image into a pre-trained monocular depth estimation network to obtain an initial monocular depth map, and performing a global linear transformation on the initial monocular depth map based on the LiDAR depth measurement values ​​at all valid pixel positions in the sparse LiDAR depth map to obtain an intermediate depth map. Within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, the depth values ​​of the intermediate depth map are optimized to generate a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, the optimization calculation makes the depth value of the locally optimized depth map at its same coordinate position approach the LiDAR depth measurement value stored in that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth value of the intermediate depth map in the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in texture information. Finally, based on the intermediate depth map and the locally optimized depth map, the final fused depth map is generated, thereby solving the problems of insufficient reliability of RGB-D cameras, large sparse blind zone of solid-state LiDAR, and blurred monocular depth scale in the existing technology.

[0022] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0023] Please see Figure 1 , Figure 1 A flowchart of an image processing method provided in an embodiment of this application is shown. It should be understood that this image processing method can be executed by an image processing device, and the specific device can be configured according to actual needs; this application embodiment is not limited thereto. For example, the image processing device can be a computer, or a server, etc. Specifically, the image processing method includes: Step S110: Simultaneously acquire RGB images captured by the RGB-D camera and point cloud data captured by the solid-state LiDAR.

[0024] Specifically, the RGB-D camera and solid-state LiDAR are controlled to trigger synchronously, acquiring the RGB image of the target scene and the original solid-state LiDAR point cloud at the current moment.

[0025] Step S120: Project the point cloud data onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image. In the sparse LiDAR depth map, successfully projected pixel positions are marked as valid pixels, and valid pixels store the corresponding LiDAR depth measurement value; pixel positions that were not successfully projected are marked as invalid pixels.

[0026] Specifically, firstly, time synchronization and data alignment can be performed. Through hardware trigger signals or software synchronization mechanisms based on precise timestamps, it is ensured that each frame of RGB image captured by the RGB-D camera corresponds strictly to the same physical moment as each frame of point cloud data captured by the solid-state LiDAR, eliminating timing misalignments caused by sensor acquisition delays or platform movement, and laying the foundation for subsequent pixel-level fusion.

[0027] Secondly, spatial calibration and coordinate transformation are performed. Before system deployment, joint calibration of the RGB-D camera and solid-state LiDAR must be completed to obtain the precise spatial transformation relationship between the two, i.e., the rotation-translation matrix (extrinsic parameter matrix) from the LiDAR coordinate system to the camera coordinate system. During data processing, the previously calibrated camera intrinsic and extrinsic parameter matrices are used to uniformly transform the original 3D point cloud acquired by the LiDAR to the camera coordinate system.

[0028] Finally, point cloud projection and sparse depth map generation are completed. 3D points in the camera coordinate system are projected onto the 2D image plane, and their corresponding pixel coordinates are calculated. For each LiDAR point successfully projected onto the image pixel coordinates, the actual depth value of that point is recorded at its corresponding position in a pre-created sparse LiDAR depth map with the same resolution as the RGB image, and this pixel is marked as a "valid pixel." Pixel positions in the image not covered by LiDAR point projections are marked as "invalid pixels," typically by assigning a special value (such as a non-numeric value). This generates a sparse LiDAR depth map that is strictly aligned with the RGB image in pixel coordinates, containing sparse but absolutely accurate depth information and valid / invalid region markings. A LiDAR point is considered 'successfully projected' if and only if it is located in front of the camera and its calculated pixel coordinates fall within the valid area of ​​the image.

[0029] Step S130: The RGB image is input into a pre-trained monocular depth estimation network to obtain an initial monocular depth map. This monocular depth estimation network is used to predict a depth value for each pixel in the RGB image.

[0030] It should be understood that the specific network of the monocular depth estimation network can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0031] For example, the monocular depth estimation network can be a convolutional neural network with an existing encoder-decoder structure, or it can be an existing visual Transformer network, etc.

[0032] Step S140: Perform a global linear transformation on the initial monocular depth map based on the lidar depth measurement values ​​at all effective pixel positions in the sparse lidar depth map to obtain an intermediate depth map.

[0033] Specifically, at all valid pixel locations of the sparse LiDAR depth map, the depth estimate of the initial monocular depth map at the same location is obtained; based on the depth estimate and the corresponding LiDAR depth measurement, a set of linear transformation parameters is determined, and the initial monocular depth map is subjected to a global linear transformation using the linear transformation parameters to obtain an intermediate depth map.

[0034] For example, firstly, in the sparse LiDAR depth map, the set of all locations marked as valid pixels is identified. For each valid pixel location in the set, a dual data readout is performed: on the one hand, the LiDAR depth measurement value stored at that location is read from the sparse LiDAR depth map, which represents the actual physical distance (usually in meters) from the camera's optical center to the corresponding scene point; on the other hand, at the exact same pixel coordinates, a depth estimate is read from the initial monocular depth map, which is a relative depth value predicted by the network, whose magnitude only indicates the order of proximity and lacks physical units.

[0035] Secondly, based on the aforementioned data set (which includes the LiDAR depth measurement and depth estimate corresponding to each effective pixel position), it is assumed that there is a global linear transformation relationship between the depth estimate and the LiDAR depth measurement, i.e., y = a*x + b, where a is the global scale scaling factor to be determined, b is the global depth offset to be determined, x is the depth estimate, and y is the LiDAR depth measurement. Parameters a and b together constitute a set of linear transformation parameters. The optimal a and b are solved using the least squares linear regression method, with the goal of making the transformed predicted value as close as possible to the actual measurement value.

[0036] Subsequently, the obtained set of linear transformation parameters is applied to each pixel in the initial monocular depth map, that is, the depth estimate of each pixel in the initial monocular depth map is substituted into the equation constructed by the set of linear transformation parameters, thereby obtaining the intermediate depth map.

[0037] Step S150: Within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, the depth values ​​of the intermediate depth map are optimized to generate a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, the optimization calculation ensures that the depth value of the locally optimized depth map at its corresponding coordinate position approaches the LiDAR depth measurement value stored for that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth values ​​of the intermediate depth map during the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in the texture information.

[0038] Specifically, within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, an optimization calculation with dual objectives is performed based on the scale-corrected intermediate depth map to generate a locally optimized depth map. Specifically: First, the primary goal of the optimization computation is to achieve a high-precision approximation of depth values ​​at the LiDAR measurement points. Specifically, for each valid pixel location marked in the sparse LiDAR depth map, a strong constraint is imposed during the optimization process. This constraint requires that the depth value of the final locally optimized depth map, at the location where it is exactly the same as the valid pixel's coordinates, must be as close as possible to the LiDAR depth measurement value stored in that valid pixel. This goal ensures that in areas where the sensor directly provides real physical measurements, the optimized depth results inherit the high-precision characteristics of the LiDAR, thereby achieving measurement-level depth accuracy in key near-field regions.

[0039] Secondly, the second objective of the optimization computation is to implement an edge-aware adaptive smoothing regularization based on the texture information of the RGB image. The core of this technique lies in the fact that its smoothing intensity is not fixed, but rather intelligently and dynamically adjusted according to the degree of local texture change in the RGB image. Furthermore, the smoothing intensity is negatively correlated with the degree of texture change; that is, the smaller the texture change (the smoother), the greater the smoothing intensity; the greater the texture change (the more drastic), the smaller the smoothing intensity. Specifically, in regions of the RGB image where color, brightness, or texture features are continuous and uniform (e.g., inside the surface of the same object), the algorithm assigns a larger smoothing weight, thus applying a strong smoothing constraint. This forces the depth values ​​of adjacent pixels within this region to converge, causing the depth map to tend towards smooth changes, effectively suppressing noise and obtaining a physically coherent surface. Conversely, in regions of the RGB image where color, brightness, or texture features undergo significant abrupt changes (e.g., the boundaries between different objects, the edges between an object and the background), due to the drastic texture change, the algorithm assigns a very small smoothing weight, which is equivalent to significantly weakening or even eliminating the smoothing constraint. This "edge-aware" weight allocation mechanism, based on the principle of "negative correlation," allows the optimization process to strictly respect the natural visual boundaries in the image. As a result, in the generated depth map, the depth contour of the object can be clearly aligned with the edges of the RGB image, perfectly preserving important geometric boundaries while maintaining the smoothness of the object's interior.

[0040] By incorporating the two objectives mentioned above (i.e., high-precision data fidelity at sparse anchor points and smooth edge performance based on image guidance across the global region) into a unified optimization framework and solving them collaboratively, the resulting locally optimized depth map achieves a balance between accuracy and visual quality. Within the effective coverage area of ​​the LiDAR, the depth values ​​in this map exhibit high reliability; simultaneously, its overall depth variation pattern is highly consistent with the visual structure of the scene, laying a solid foundation for subsequent fusion and final application.

[0041] Step S160: Generate the final fused depth map based on the intermediate depth map and the locally optimized depth map.

[0042] Specifically, based on the markings of the sparse LiDAR depth map, a final fused depth map is generated; where, for each location marked as an invalid pixel, its depth value is taken from the corresponding location in the intermediate depth map; and for each location marked as a valid pixel, its depth value is taken from the corresponding location in the local optimized depth map.

[0043] For example, its implementation follows a clear branching logic that relies entirely on the pixel labels of the generated sparse LiDAR depth map. Each pixel location on the entire image plane can be divided into two mutually exclusive sets based on its labeling status in this sparse map: an invalid pixel set and a valid pixel set. For pixels in these two sets, distinct but purposefully defined depth value assignment strategies are adopted: First, for each image location marked as an "invalid pixel," this indicates that the solid-state LiDAR did not successfully measure data at that location, falling within the sensor's blind spot (e.g., outside the ranging range, outside a narrow field of view, or corresponding to a low-reflectivity surface). For these locations, the system determines that direct accuracy constraints from the LiDAR cannot be obtained. Therefore, the depth values ​​of the final fused depth map at these locations will be directly taken from the corresponding locations in the intermediate depth map. The essence of this strategy is to fully trust and adopt the prediction results of the monocular depth estimation network after global scale correction in areas "unseen" by the sensor. Since the intermediate depth map has been given the correct physical scale through linear transformation, and the monocular network itself has strong scene understanding and completion capabilities, the resulting depth values, although not comparable to the measurement points in absolute accuracy, provide a "best reasonable estimate" that conforms to the overall geometry and semantic priors of the scene at the correct physical scale, thus elegantly filling the perception gaps.

[0044] Secondly, for each image location marked as a "valid pixel," this indicates that the solid-state LiDAR provides a high-confidence direct depth measurement at that location. These locations are valuable accuracy anchors. For these locations, the system determines that the more accurate depth information, after local optimization processing, should be prioritized. Therefore, the depth values ​​of the final fused depth map at these locations will be taken from the corresponding locations of the locally optimized depth map. The essence of this strategy is to adopt the result after dual optimization of "data fidelity" and "edge smoothing" where the sensor can "see clearly." This result, while strictly anchored to the true LiDAR measurement value, also achieves a more visually reasonable and smoother surface structure through image-guided optimization, thus achieving a unity of measurement-level accuracy and high visual quality within the effective area of ​​the sensor.

[0045] Through the aforementioned label-based, deterministic pixel-level assignment operations, the system successfully and seamlessly combines the "complete coverage advantage of monocular prediction after global scale correction" with the "high precision advantage of LiDAR anchor point regions after local optimization." The final output fused depth map is a dense depth map with complete resolution and a definite depth value for each pixel. It not only has extremely high accuracy and clear object boundaries in the near-field region covered by LiDAR, but also provides scale-consistent depth estimation that conforms to scene understanding in the far-field and blind spots not covered by LiDAR. This achieves coherent and reliable 3D environment perception from near to far and from measured to unmeasured areas, providing high-quality input for downstream tasks such as robot navigation, obstacle avoidance, and 3D reconstruction.

[0046] In addition, the image processing method also includes: performing reliability verification on the fused depth map; wherein the reliability verification includes at least one of the following: consistency verification based on continuous frames and sensor motion information, consistency verification based on lidar measurement data, and verification based on the continuity of the depth map itself.

[0047] For consistency verification based on continuous frames and sensor motion information, the core principle is the rigid body motion invariance in the physical world. When a robot platform equipped with sensors moves, the 3D coordinate sequence formed by static scene points in the fused depth map of multiple consecutive frames should have a trajectory that is strictly consistent with the platform's own motion trajectory obtained by sensors such as inertial measurement units and wheeled odometers. In practice, the system extracts the 3D position of the same static feature point from continuous frames, calculates its displacement change, and compares it with the displacement measured by the platform's sensors within the same time period. If the deviation exceeds a preset threshold, it indicates that the depth estimation may have systematic errors in scale or dynamic object segmentation, triggering a reliability alarm. This verification is mainly used to capture temporal inconsistencies caused by scale drift, misjudgment of dynamic objects, etc. For consistency verification based on LiDAR measurement data, the core principle of this verification is sensor cross-validation. During the generation process, the fused depth map has already been subject to optimization constraints at the effective measurement points of the LiDAR. This verification quantitatively evaluates the final achievement of these constraints through post-hoc statistical methods. Specifically, the system compares the depth values ​​of the fused depth map with the original LiDAR depth measurements point-by-point at all effective pixel locations of the sparse LiDAR depth map, calculating statistical indicators such as average error and root mean square error. If the error exceeds a reasonable threshold set based on sensor accuracy, it indicates that the local optimization process may not have converged sufficiently, or that it has been severely disturbed in some areas, causing the result to fail to effectively anchor to the high-precision true value. This verification directly measures the absolute accuracy achieved by the fused depth map within the known true value region.

[0048] For verification based on the continuity of the depth map itself, the core principle is the physical continuity of the scene surface. In the real world, except for object boundaries, the depth changes inside the object surface are usually smooth and continuous; at the boundary between the effective and ineffective areas of the LiDAR, the depth should not exhibit non-physical discontinuities that do not conform to the extension law of the object surface. This verification checks the geometric rationality by analyzing the spatial gradient or depth distribution of the fused depth map itself. Specifically, the system focuses on checking two types of regions: first, the interior of the object surface (which can be initially judged through RGB image segmentation), where the depth gradient should be small; second, the boundary between the effective and ineffective areas of the LiDAR, where the depth change should be smooth or aligned with the image edge. If an abnormally large depth jump is detected inside the object, or unreasonable depth discontinuities are found at the region boundary, it indicates that the fusion process may have introduced noise or produced unnatural stitching marks. This verification is used to ensure that the geometric structure of the output depth map conforms to physical laws and visual common sense.

[0049] By implementing one or more of the above verifications, the system can evaluate the quality of the fused depth map from multiple perspectives, promptly detect potential estimation biases, inconsistencies, or geometric anomalies, thereby providing confidence references for downstream applications or triggering corresponding error handling mechanisms, significantly improving the security and reliability of the entire sensing system.

[0050] In summary, by utilizing the aforementioned technical solutions, this application seamlessly combines the high-precision measurement of short-range LiDAR with the reasonable estimation of long-range monocular vision, overcoming the physical limitations of a single sensor. Furthermore, this application solves the scale ambiguity problem in monocular depth estimation by leveraging the absolute scale constraint of LiDAR, and provides physical measurement-level accuracy guarantees within the sensor's effective area, thereby improving the accuracy and reliability of depth estimation. Moreover, even in areas where LiDAR fails due to distance, reflectivity, or other factors, the system can still provide meaningful depth information, achieving graceful degradation and meeting the high safety requirements of robots, thus enhancing the system's robustness.

[0051] Furthermore, the output fused depth map is more conducive to subsequent high-level vision tasks such as 3D object detection and semantic segmentation than sparse point clouds, thereby improving scene understanding capabilities.

[0052] It should be understood that the above image processing method is merely exemplary, and those skilled in the art can make various modifications based on the above method, and the modified solutions also fall within the protection scope of this application.

[0053] Please see Figure 2 , Figure 2A structural block diagram of an image processing apparatus 200 provided in an embodiment of this application is shown. It should be understood that the image processing apparatus 200 is capable of performing the various steps in the above method embodiments. The specific functions of the image processing apparatus 200 can be found in the description above; detailed descriptions are omitted here to avoid repetition. The image processing apparatus 200 includes at least one software function module that can be stored in memory or embedded in the operating system (OS) of the image processing apparatus 200 in the form of software or firmware. Specifically, the image processing apparatus 200 includes: The acquisition module 210 is used to simultaneously acquire RGB images captured by an RGB-D camera and point cloud data captured by a solid-state LiDAR. The first generation module 220 is used to project point cloud data onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image; wherein, in the sparse LiDAR depth map, the pixel positions that are successfully projected are marked as valid pixels, and the valid pixels store the corresponding LiDAR depth measurement values, while the pixel positions that are not successfully projected are marked as invalid pixels. The input module 230 is used to input the RGB image into a pre-trained monocular depth estimation network to obtain the output initial monocular depth map; The global linear transformation module 240 is used to perform a global linear transformation on the initial monocular depth map based on the lidar depth measurement values ​​at all effective pixel positions in the sparse lidar depth map to obtain an intermediate depth map. The optimization calculation module 250 is used to optimize the depth values ​​of the intermediate depth map within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, generating a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, in the optimization calculation, the depth value of the locally optimized depth map at its same coordinate position approaches the LiDAR depth measurement value stored in that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth values ​​of the intermediate depth map in the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in the texture information. The second generation module 260 is used to generate the final fused depth map based on the intermediate depth map and the locally optimized depth map.

[0054] In one possible embodiment, the monocular depth estimation network is a convolutional neural network with an encoder-decoder structure or a visual Transformer network.

[0055] In one possible embodiment, the global linear transformation module 240 is configured to: obtain depth estimates of the initial monocular depth map at the same locations at all valid pixel locations of the sparse LiDAR depth map; determine a set of linear transformation parameters based on the depth estimates and the corresponding LiDAR depth measurements; and perform a global linear transformation on the initial monocular depth map using the linear transformation parameters to obtain an intermediate depth map.

[0056] In one possible embodiment, the second generation module 260 is specifically configured to: generate a final fused depth map based on the markings of the sparse lidar depth map; wherein, for each location marked as an invalid pixel, its depth value is taken from the corresponding location of the intermediate depth map; and for each location marked as a valid pixel, its depth value is taken from the corresponding location of the local optimized depth map.

[0057] In one possible embodiment, the image processing apparatus 200 further includes: A reliability verification module (not identified) is used to verify the reliability of the fused depth map; wherein, the reliability verification includes at least one of the following: consistency verification based on continuous frames and sensor motion information, consistency verification based on lidar measurement data, and verification based on the continuity of the depth map itself.

[0058] Since the apparatus described in the above embodiments of the present invention is an apparatus used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the methods described in the above embodiments of the present invention, and therefore will not be described again here. All apparatuses used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0061] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0062] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0063] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0064] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. An image processing method, characterized in that, include: Simultaneously acquire RGB images captured by an RGB-D camera and point cloud data captured by a solid-state LiDAR; The point cloud data is projected onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image; wherein, in the sparse LiDAR depth map, the pixel positions that are successfully projected are marked as valid pixels, and the valid pixels store the corresponding LiDAR depth measurement values, while the pixel positions that are not successfully projected are marked as invalid pixels. The RGB image is input into a pre-trained monocular depth estimation network to obtain the initial monocular depth map output. Based on the lidar depth measurement values ​​at all effective pixel positions in the sparse lidar depth map, a global linear transformation is performed on the initial monocular depth map to obtain an intermediate depth map. Within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, the depth values ​​of the intermediate depth map are optimized to generate a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, in the optimization calculation, the depth value of the locally optimized depth map at its same coordinate position approaches the LiDAR depth measurement value stored in that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth values ​​of the intermediate depth map in the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change in the texture information. Based on the intermediate depth map and the local optimized depth map, the final fused depth map is generated.

2. The image processing method according to claim 1, characterized in that, The monocular depth estimation network is a convolutional neural network with an encoder-decoder structure or a visual Transformer network.

3. The image processing method according to claim 1, characterized in that, The process of performing a global linear transformation on the initial monocular depth map based on the lidar depth measurements at all effective pixel locations in the sparse lidar depth map to obtain an intermediate depth map includes: At all valid pixel locations of the sparse lidar depth map, obtain the depth estimate of the initial monocular depth map at the same location; Based on the depth estimate and the corresponding lidar depth measurement, a set of linear transformation parameters are determined, and the initial monocular depth map is subjected to a global linear transformation using the linear transformation parameters to obtain the intermediate depth map.

4. The image processing method according to any one of claims 1 to 3, characterized in that, The step of generating the final fused depth map based on the intermediate depth map and the locally optimized depth map includes: Based on the markings in the sparse lidar depth map, the final fused depth map is generated; wherein, for each position marked as an invalid pixel, its depth value is taken from the corresponding position in the intermediate depth map; for each position marked as a valid pixel, its depth value is taken from the corresponding position in the local optimized depth map.

5. The image processing method according to claim 1, characterized in that, The image processing method further includes: The reliability of the fused depth map is verified; wherein the reliability verification includes at least one of the following: consistency verification based on continuous frames and sensor motion information, consistency verification based on lidar measurement data, and verification based on the continuity of the depth map itself.

6. An image processing apparatus, characterized in that, include: The acquisition module is used to simultaneously acquire RGB images captured by an RGB-D camera and point cloud data captured by a solid-state LiDAR. The first generation module is used to project the point cloud data onto the image coordinate system corresponding to the RGB image to generate a sparse LiDAR depth map aligned with the pixels of the RGB image; wherein, in the sparse LiDAR depth map, the pixel positions that are successfully projected are marked as valid pixels, and the valid pixels store the corresponding LiDAR depth measurement values, while the pixel positions that are not successfully projected are marked as invalid pixels. The input module is used to input the RGB image into a pre-trained monocular depth estimation network to obtain an initial monocular depth map output. The global linear transformation module is used to perform a global linear transformation on the initial monocular depth map based on the lidar depth measurement values ​​at all effective pixel positions in the sparse lidar depth map to obtain an intermediate depth map. An optimization calculation module is used to optimize the depth values ​​of the intermediate depth map within the image region corresponding to the effective pixels identified by the sparse LiDAR depth map, generating a locally optimized depth map. The optimization calculation is configured to simultaneously satisfy the following two objectives: for each effective pixel in the sparse LiDAR depth map, in the optimization calculation, the depth value of the locally optimized depth map at its same coordinate position approaches the LiDAR depth measurement value stored in that effective pixel; based on the texture information of the RGB image, a smoothing constraint is applied to the depth values ​​of the intermediate depth map in the optimization calculation, and the strength of the smoothing constraint is negatively correlated with the degree of change of the texture information. The second generation module is used to generate the final fused depth map based on the intermediate depth map and the local optimized depth map.

7. The image processing apparatus according to claim 6, characterized in that, The monocular depth estimation network is a convolutional neural network with an encoder-decoder structure or a visual Transformer network.

8. The image processing apparatus according to claim 6, characterized in that, The global linear transformation module is used to: obtain depth estimates of the initial monocular depth map at the same positions at all valid pixel positions of the sparse lidar depth map; determine a set of linear transformation parameters based on the depth estimates and the corresponding lidar depth measurements; and perform a global linear transformation on the initial monocular depth map using the linear transformation parameters to obtain the intermediate depth map.

9. The image processing apparatus according to any one of claims 6 to 8, characterized in that, The second generation module is specifically used to: generate the final fused depth map based on the markings of the sparse lidar depth map; wherein, for each position marked as an invalid pixel, its depth value is taken from the corresponding position of the intermediate depth map; for each position marked as a valid pixel, its depth value is taken from the corresponding position of the local optimized depth map.

10. The image processing apparatus according to claim 6, characterized in that, The image processing device further includes: A reliability verification module is used to perform reliability verification on the fused depth map; wherein the reliability verification includes at least one of the following: consistency verification based on continuous frames and sensor motion information, consistency verification based on lidar measurement data, and verification based on the continuity of the depth map itself.

Images