Cross-modal binocular vision depth estimation method based on contrast learning

Abstract

A cross-modal binocular vision depth estimation method based on contrast learning comprises the following steps: collecting cross-modal image data of monocular alignment, collecting RGB images and non-RGB images through a multi-modal camera system, and performing strict camera calibration to realize alignment of pixel levels; a cross-modal binocular data generation model is constructed, binocular image data meeting the standard is generated based on cross-modal image data of monocular alignment, and a parallax transformation and edge perception restoration module based on depth is included; constructing a cross-modal binocular depth estimation model, introducing feature pre-training and supervised constraint optimization based on comparative learning, and training by using an aligned monocular cross-modal data set and a generated cross-modal binocular data set; and storing the training parameters, and generating a parallax image according to the input cross-modal binocular data. According to the invention, the accuracy of the binocular depth estimation model is improved by generating the cross-modal data, and the stability of the cross-modal depth estimation method is improved from the perspective of the model and the generated data.

Description

Technical Field

[0001] This invention relates to a cross-modal binocular vision depth estimation method based on contrastive learning, belonging to the fields of computer vision and deep learning technology. Background Technology

[0002] Stereo matching is a core task in computer vision, aiming to obtain depth information of a scene by calculating the disparity between two input images. This technology plays a crucial role in many downstream tasks such as robot navigation, autonomous driving, and augmented reality. With the rapid development of deep learning methods, learning-based stereo matching algorithms have achieved significant performance improvements. However, although existing stereo matching models perform very well in normal environments, their performance remains significantly limited in extreme scenarios (such as low-light and high dynamic range scenes).

[0003] With the continuous development of imaging technology, some researchers have attempted to use non-RGB imaging methods (such as infrared cameras and event cameras) to acquire image data. These imaging methods have unique imaging characteristics. Although they lose color and texture information compared to RGB imaging, they exhibit stronger robustness in special scenarios such as low light, avoiding significant image quality degradation. Therefore, researchers in the field of stereo matching have begun to explore the use of these non-RGB data for stereo matching to address the problem of poor performance of RGB stereo matching in special scenarios.

[0004] While these methods have improved stereo matching performance in special environments to some extent, their subsequent applications are limited because non-RGB imaging methods cannot capture the color and texture information of the scene. Furthermore, under normal lighting conditions, the stereo matching performance of these methods is far inferior to that of RGB images. To simultaneously consider the color and texture information of RGB and the imaging advantages of non-RGB in special environments, researchers have proposed cross-modal stereo matching methods. For example, RGB images are used as the left-eye input and infrared images as the right-eye input to obtain the disparity between the two modalities. However, cross-modal stereo matching faces several major challenges. Existing stereo matching methods typically employ shared encoder feature extractors, but the features of images from different modalities differ significantly, making this method unsuitable for cross-modal stereo matching. Additionally, due to limitations in acquiring cross-modal data caused by high imaging conditions and modal alignment issues, the number of currently available cross-modal datasets is limited, restricting the generalization ability and performance of the models.

[0005] Therefore, the key to solving the above problems lies in how to construct an efficient and flexible cross-modal binocular vision depth estimation method. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies by creatively proposing a cross-modal binocular vision depth estimation method and system based on contrastive learning.

[0007] The present invention is achieved using the following technical solution.

[0008] A cross-modal binocular vision depth estimation method based on contrastive learning includes the following steps:

[0009] Step 1: Collect monocular aligned cross-modal image data based on the definition and acquisition method of cross-modal images.

[0010] Specifically, a calibrated multimodal monocular camera system (e.g., a system consisting of RGB and infrared cameras) is used to acquire cross-modal monocular image pairs (e.g., RGB and infrared images) under normal conditions. To ensure that the two modal images are aligned at the pixel level, the multimodal camera needs to be rigorously calibrated to obtain the camera's intrinsic and extrinsic parameters. The calibration process includes camera intrinsic parameter calibration, extrinsic parameter calibration, and image alignment steps.

[0011] First, by calibrating the intrinsic parameters of the camera for each mode, the camera's intrinsic parameter matrix can be obtained:

[0012]

[0013] Among them, (f x ,f y ) is the focal length of the camera in the (x) and (y) directions, (c) x ,c y ) are the coordinates of the principal point of the image.

[0014] Secondly, by calibrating the extrinsic parameters of the multimodal camera, the rotation matrix (R) and translation vector (T) between the two cameras can be obtained:

[0015] R PGB =R·P MOD +T#(2)

[0016] Among them, (P) RGB ) and (P MOD The coordinates of the RGB camera and the non-RGB camera in the world coordinate system are represented by , respectively. The rotation matrix (R) describes the rotation relationship between the two cameras, and the translation vector (T) describes the spatial displacement between the two cameras.

[0017] Finally, using the calibrated camera intrinsic and extrinsic parameters, non-RGB images can be projected onto the coordinate system of RGB images, achieving cross-modal image alignment. The projection formula is:

[0018] p′=K·[R|T]·P#(3)

[0019] Where (P) are the 3D points under the non-RGB camera, (p′) are the pixels projected onto the RGB image plane, (K) is the intrinsic parameter matrix of the RGB camera, and ([R|T]) is the extrinsic parameter matrix from non-RGB to RGB camera. Finally, we obtain the monocular aligned RGB image I. RGB and cross-modal image I cross .

[0020] By following the steps above, we can ensure pixel-level alignment of cross-modal monocular images, providing high-quality registration data for subsequent cross-modal tasks.

[0021] Step 2: Construct a cross-modal stereo data generation model, introduce depth-based disparity transformation, and use monocular aligned cross-modal image data to generate standard-compliant stereo image data.

[0022] Specifically, using monocular-aligned cross-modal image data, a cross-modal binocular data generation model is constructed to generate binocular image data that conforms to real-world scenes. This model generates disparity maps through depth-based transformations and combines them with an edge-aware inpainting module to enhance the realism of the generated images.

[0023] Step 2.1: Construct a cross-modal stereo data generation model based on the generated depth disparity map.

[0024] Specifically, the cross-modal stereo data generation model first generates a disparity map through disparity definition and depth estimation, thereby generating the pixel correspondence between the left and right views. The disparity definition is as follows:

[0025] D(i) = x l (i)-x r (i′)#(4)

[0026] Where D(i) is the disparity value of pixel i, x l (i) and x r (i′) are the horizontal coordinates of pixels i and i′ in the left and right views, respectively.

[0027] Disparity map D generated using a monocular depth estimation model mono The disparity map values ​​generated by the depth estimation model range from [0,1]. To generate a pixel-level disparity map D... mono ', We are with D mono The scaling conversion is performed using the following formula:

[0028] D′ mono =αD mono #(5)

[0029] Where α is the scaling factor, α∈[d min ,d max ];dmin and d max These are the minimum and maximum scaling values, respectively. To increase the diversity of the data, α is derived from a uniform distribution U(d min ,d max Random sampling is performed in the sample.

[0030] In this way, the generated disparity map can effectively describe the correspondence between pixels in the left and right views and provide accurate geometric information for the subsequent image generation process.

[0031] Then, using pixel-level disparity maps D mono 'By combining the monocular aligned cross-modal data collected in step 1, we constructed a cross-modal binocular data generation model.'

[0032] Specifically, we use D′ mono Aligning RGB images in monocular data using disparity mapping relationships RGB Mapped to With cross-modal image I cross

[0033] Step 2.2: Based on edge perception technology, construct an edge perception repair module to fill in image holes caused by parallax mapping.

[0034] During the disparity map generation and pixel mapping process, holes may appear in the right-view image due to occlusion or viewing angle differences. To address this issue, an edge-aware repair module was built to repair invisible areas and enhance the realism of the generated image.

[0035] First, the edge-aware inpainting module detects object edge information and uses a stable diffusion model to repair occluded areas. First, based on the disparity map D... mono Generate edge masks M∈[0,1] H×W The formula for generating the edge mask is:

[0036]

[0037] in, The horizontal gradient of the disparity map is represented by τ, which is the threshold used for edge detection. Through edge masking, the edge-aware inpainting module can effectively preserve the edge information of objects and reduce the fusion problem between foreground and background.

[0038] During the restoration process, the edge-aware restoration module selects a portion of background pixels and performs pixel mapping with foreground pixels. Simultaneously, it combines an edge mask M to guide a stable diffusion model to restore occluded areas. The specific steps are as follows:

[0039] First, edge pixels are detected and moved to alleviate the blending problem between the foreground and background.

[0040] Secondly, a stable diffusion model is used to repair the voided areas, generating a complete right-view image.

[0041] By introducing an edge-aware repair module, the generated right-view image is visually more realistic and can effectively preserve the depth structure and edge details of objects.

[0042] This step utilizes depth-based disparity generation and edge-aware inpainting modules to generate high-quality binocular image data from monocular-aligned cross-modal image data. The generated images exhibit high geometric consistency and visual realism, providing reliable data support for cross-modal stereo matching tasks.

[0043] Step 3: Construct a cross-modal stereo depth estimation model, introduce feature pre-training based on contrastive learning, and train it using the aligned monocular cross-modal dataset and the generated cross-modal stereo dataset.

[0044] Specifically, a cross-modal stereo depth estimation model is constructed, employing a feature pre-training strategy based on contrastive learning, and trained by combining an aligned monocular cross-modal dataset and a generated cross-modal stereo dataset. The model consists of two parts: feature pre-training based on contrastive learning (step 3.1) and depth estimation optimized through supervised constraints (step 3.2).

[0045] Step 3.1 Establish contrastive learning constraints and optimize network parameters using a loss function.

[0046] To address the discrepancy between RGB and non-RGB modal feature extraction in cross-modal stereo tasks, a parallel stream encoder architecture was designed. This architecture ensures that data features from different modalities do not interfere with each other through independent feature encoding paths.

[0047] For the RGB image encoder, we use MobileNetV2 as the backbone network and input image I r Downsampled to 1 / 32 of the original size, then restored to 1 / 4 scale through an upsampling module and skip connections to generate multi-scale features:

[0048]

[0049] Among them, {f r,i} is a multi-scale feature; It is a feature encoder for RGB images.

[0050] For non-RGB modes (such as thermal infrared, near-infrared, or polarized images), a mode-specific preprocessing transformation is first performed. Convert non-RGB images to uniform format. The format is then used to generate multi-scale features via a non-RGB modality feature encoder:

[0051]

[0052] in, It is a multi-scale feature of non-RGB modality; It is a feature encoder for non-RGB images.

[0053] The features generated by the feature encoder will be used to construct the cost volume, and then used by the subsequent depth estimation module to generate disparity predictions.

[0054] To mitigate imaging differences across modalities (e.g., RGB images contain color information, while NIR / IR images contain reflection or thermal imaging information), a feature pre-training strategy based on contrastive learning is designed to align representations of different modalities in the feature space.

[0055] In the contrastive learning pre-training phase, only cross-modal monocular image pairs are needed (no need for binocular image pairs and disparity information). N image patches are obtained through random cropping, and then these patches from different modalities are input into the parallel stream encoder to generate paired feature embeddings.

[0056]

[0057] in, F is the feature embedding generated in the non-RGB modality; N is the feature embedding generated in the RGB modality; and N is the number of randomly cropped image patches.

[0058] Then, the contrastive learning loss is calculated based on the embedded features, maximizing the cosine similarity of the true paired features and minimizing the similarity of other incorrect pairs. The symmetric cross-entropy loss function for contrastive learning is defined as follows:

[0059]

[0060] in, is the cosine similarity matrix of the feature embedding; N2(·) denotes L2 normalization; G is the label matrix used to calculate the cross-entropy loss, with the elements on the diagonal being positive samples and the rest being negative samples; It is the cross-entropy loss function.

[0061] When calculating the contrastive learning loss, image patches at the same location are considered positive samples, while other image patches are considered negative samples.

[0062] By using contrastive learning for feature pre-training, the model can learn unified feature representations across modalities without disparity supervision, thereby enhancing the robustness of the cross-modal binocular depth estimation model.

[0063] Step 3.2 Establish supervised constraints and optimize network parameters using the loss function.

[0064] The generated multi-scale features are used to construct the cost volume, and then used by the subsequent depth estimation module to generate the initial disparity prediction P0 and the final disparity prediction P. i The specific process is as follows:

[0065] Using cost volume building blocks and Hourglass networks Generate initial disparity prediction:

[0066]

[0067] The initial disparity prediction is iteratively optimized using the ConvGRU strategy to generate the final disparity prediction:

[0068]

[0069] The network parameters are further optimized through supervised constraints. The loss function consists of contrastive loss. and depth estimation loss The composition includes a depth estimation loss that measures the difference between predicted and true disparity, comprising L1... Weight of slip loss.

[0070] The final total loss is defined as:

[0071]

[0072] This step involves constructing a cross-modal stereo depth estimation model based on contrastive learning-based feature pre-training and supervised constraint optimization. The model is trained using aligned monocular cross-modal datasets and generated cross-modal stereo datasets. The model aligns inter-modal representations in the feature space and improves disparity prediction accuracy through supervised learning, providing a high-quality solution for cross-modal stereo vision tasks.

[0073] Step 4: Save the training parameters, generate a binocular matching disparity image based on the input cross-modal binocular data, and complete the inference and index evaluation.

[0074] Specifically, in order to objectively evaluate the quality of the generated natural images, objective evaluation metrics can be generated based on EPE (End-Point Error) and D1 (Disparity One Pixel Error).

[0075] Based on the above method, this invention further proposes a stereo depth estimation system based on cross-modal data, including a monocular cross-modal data collection subsystem, a cross-modal data generation subsystem, a cross-modal stereo depth estimation subsystem, and a supervised optimization and result evaluation subsystem.

[0076] The connections between the above components are as follows:

[0077] The output of the monocular cross-modal data collection subsystem is connected to the input of the cross-modal data generation subsystem and the cross-modal binocular depth estimation subsystem. The output of the cross-modal data generation subsystem is connected to the input of the cross-modal binocular depth estimation subsystem. The output of the cross-modal binocular depth estimation subsystem is connected to the input of the supervised optimization and result evaluation subsystem.

[0078] Beneficial effects

[0079] Compared with the prior art, the present invention has the following advantages:

[0080] 1. This invention introduces a cross-modal data generation method based on depth control and edge-aware inpainting, enabling the generation of high-quality cross-modal binocular image data in complex scenes (such as under conditions of insufficient lighting, occlusion, or significant differences in imaging between modalities). The generated data exhibits high realism in depth and geometry, providing rich and reliable training data for cross-modal binocular depth estimation models, thereby significantly improving the model's adaptability and robustness in various complex environments.

[0081] 2. This invention designs a parallel encoder architecture to extract features independently for RGB and non-RGB modalities, ensuring that the features of the two modalities do not interfere with each other. By combining contrastive learning constraints and supervised optimization constraints, the model can effectively solve the inconsistency problem caused by the feature differences between modalities, while ensuring the accuracy and geometric consistency of disparity prediction, thereby improving the model's performance in complex modal scenarios.

[0082] 3. This invention avoids reliance on binocular disparity supervision in the early stages of training by pre-training cross-modal features during the contrastive learning phase, significantly reducing the need for manually labeled disparity data. Training with generated cross-modal data lowers the cost of data acquisition and labeling. Simultaneously, the efficient encoder design and contrastive learning strategy significantly improve the model's training efficiency, enabling the model to achieve high accuracy and robustness even with limited computational resources. Attached Figure Description

[0083] Figure 1 This is a flowchart of the method of the present invention.

[0084] Figure 2 This is a schematic diagram of the data generation and neural network structure described in the method of the present invention.

[0085] Figure 3 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0086] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below in conjunction with the accompanying drawings and examples.

[0087] Example

[0088] like Figure 1 As shown, a stereo depth estimation method based on cross-modal data includes the following steps:

[0089] Step 1: Collect monocular cross-modal alignment datasets and real binocular cross-modal datasets to provide foundational data for model training. Through rigorous calibration and modal registration methods, ensure pixel-level alignment of RGB and non-RGB modalities (such as infrared, thermal infrared, polarized images, etc.) to improve data diversity and reliability.

[0090] Step 2: Construct a cross-modal stereo data generation model, introduce a depth-based transformation module and an edge-aware inpainting module to generate a high-quality cross-modal stereo image dataset, thereby enhancing the model's adaptability to modal differences and complex scenes.

[0091] Step 2.1: The depth-based disparity map generation module generates a disparity map using monocular depth estimation. Combined with random scaling and transformation operations, it generates initial left and right view pairing data to ensure that the generated image is consistent with the real data in the depth dimension.

[0092] Step 2.2: Introduce an edge-aware repair module to repair the hole areas in the generated image caused by occlusion or viewpoint differences, while preserving the object edge information to ensure consistency in geometric structure and depth information between the left and right images.

[0093] Step 3: Construct a cross-modal binocular depth estimation model, introduce a feature pre-training strategy and supervision constraints based on contrastive learning, optimize network parameters, and improve the consistency of feature representation and depth estimation accuracy of the model across different modal data.

[0094] Step 3.1: Comparative learning feature pre-training. By aligning the features of RGB and non-RGB modalities on a monocular cross-modal dataset, a unified feature representation between modalities is learned, thereby improving the robustness of the model.

[0095] Step 3.2: Supervised training. The network is optimized on the generated cross-modal stereo dataset and real disparity data. The accuracy of disparity prediction is improved by using supervised loss functions (such as L1 loss and smoothing constraint loss).

[0096] Step 4: Save the optimized network parameters, generate a disparity map based on the input cross-modal binocular image pairs, and combine quality evaluation metrics (such as disparity accuracy, reconstruction error, etc.) to infer and evaluate the model performance, thus completing the depth estimation task in complex scenes.

[0097] One embodiment of this application presents a cross-modal binocular vision depth estimation method based on contrastive learning. This method constructs a cross-modal binocular dataset through a monocular cross-modal data collection and generation module; optimizes network parameters using a contrastive learning feature pre-training strategy and supervised constraints; saves the training parameters; and generates cross-modal binocular matching disparity images based on the input data to complete inference and performance evaluation. This addresses the accuracy issues of stereo matching in complex scenes and modal differences in existing technologies, effectively improving the robustness and accuracy of depth estimation models in complex modalities.

[0098] Furthermore, in one specific embodiment of this application, the present invention acquires aligned cross-modal monocular image pairs (such as RGB images and infrared images), generates disparity maps using a depth estimation network, and then constructs a cross-modal binocular image dataset using a disparity generation module. A high-quality dataset is generated using a depth-to-disparity conversion formula. The disparity and depth conversion formulas are uniformly expressed as:

[0099]

[0100] Where d is the disparity, f is the camera's focal length, B is the camera baseline (i.e., the distance between the two cameras), and Z is the depth value. This conversion method yields accurate disparity labels, providing reliable data support for subsequent stereo matching model training.

[0101] Specifically, the acquisition process uses a calibrated multimodal monocular camera system (such as a system consisting of an RGB and an infrared camera) to acquire cross-modal monocular image pairs under normal conditions. To ensure that the two modal images are aligned at the pixel level, the multimodal camera needs to be rigorously calibrated, including intrinsic parameter calibration, extrinsic parameter calibration, and image alignment steps.

[0102] Internal parameter calibration is expressed by the following formula:

[0103]

[0104] Among them, f x and f y It is the focal length of the camera in the x and y directions, c x and c y These are the coordinates of the principal point of the image.

[0105] Extrinsic parameter calibration is used to obtain the rotation matrix R and translation vector T between two cameras:

[0106] P RGB =R·P MOD +T

[0107] Among them, P RGB and P MODThese represent the coordinates of the RGB camera and the non-RGB camera in the world coordinate system, respectively. Finally, the non-RGB image is projected onto the RGB image coordinate system using intrinsic and extrinsic parameters, achieving cross-modal image alignment. The projection formula is as follows:

[0108] p'=K·[R|T]·P

[0109] Where P represents a 3D point from a non-RGB camera, p' represents a pixel projected onto the RGB image plane, K is the intrinsic parameter matrix of the RGB camera, and [R|T] is the extrinsic parameter matrix from non-RGB to RGB camera. The final result is an aligned RGB image I. RGB and cross-modal image I cross .

[0110] Furthermore, in one embodiment of this application, the depth-controlled cross-modal image generation module is used to guide the generation of cross-modal stereo images using depth information. Specifically, the cross-modal stereo data generation model generates a disparity map based on depth information, constructing the correspondence between pixels in the left and right views. The definition of disparity is as follows:

[0111] D(i) = x l (i)-x r (i′)

[0112] Where D(i) is the disparity value of pixel i, x l (i) and x r (i′) are the horizontal coordinates of pixels i and i′ in the left and right views, respectively.

[0113] Disparity map D is generated using a monocular depth estimation model. mono And scale and transform it to generate a pixel-level disparity map D. mono ':

[0114] D mono ′=α·Dmono

[0115] Where α is the scaling factor for random sampling, α∈[d min ,d max In this way, the generated disparity map can effectively describe the pixel correspondence between the left and right views.

[0116] During the generation process, an edge-aware inpainting module is used to repair holes caused by parallax mapping, enhancing the realism of the generated image. The edge-aware inpainting module generates an edge mask M by detecting object edge information and then uses a stable diffusion model to repair occluded areas. The formula for generating the edge mask is:

[0117]

[0118] in, The horizontal gradient of the disparity map is represented by τ, which is the threshold for edge detection. Through the edge-aware inpainting module, the generated right-view image is visually more realistic and preserves the depth structure and edge details of objects.

[0119] Furthermore, in one embodiment of this application, a cross-modal stereo depth estimation model is constructed, introducing a feature pre-training strategy based on contrastive learning, and trained by combining an aligned monocular cross-modal dataset and a generated cross-modal stereo dataset. The model comprises two parts: feature pre-training based on contrastive learning and depth estimation optimized through supervised constraints.

[0120] During the contrastive learning pre-training phase, the model generates image patches through random cropping and generates paired feature embeddings through a parallel encoder:

[0121]

[0122] in, F is the feature embedding generated in the non-RGB modality, and F is the feature embedding generated in the RGB modality. The contrastive learning loss maximizes the cosine similarity of the true paired features and minimizes the similarity of the incorrectly paired features. The contrastive loss function is defined as follows:

[0123]

[0124] Where M is the cosine similarity matrix of the feature embedding, and G is the label matrix.

[0125] By optimizing depth estimation through supervised constraints, the model generates an initial disparity prediction P0 and a final disparity prediction P1. i And the L1 loss function is used to measure the difference between the predicted disparity and the true disparity:

[0126]

[0127] in, It is a true parallax map. λ is the disparity gradient, and λ is the smoothness constraint weight.

[0128] Furthermore, in one embodiment of this application, in order to evaluate the effect of the generated image, objective evaluation results can be generated based on metrics such as EPE (End-Point Error) and D1 (Disparity One Pixel Error).

[0129] Figure 2This diagram illustrates the neural network structure used in this application's embodiments. A stereo image dataset under cross-modal conditions is generated through a defined depth control model. The depth control module learns depth information, generates a pixel-level disparity map, and generates pixel correspondences between the left and right views through disparity mapping, ensuring that the depth of the generated image remains consistent with the true depth under cross-modal conditions. Simultaneously, an edge-aware repair module repairs image holes that occur during disparity mapping, using edge-aware technology to preserve object edge information and repair occluded areas, enhancing the visual realism and accuracy of the depth structure of the generated image. This solves the problem of image loss due to viewing angle differences in stereo visual depth estimation under cross-modal conditions. Furthermore, this application's embodiments design a stereo depth estimation network based on contrastive learning and a parallel encoder to address the feature inconsistency problem caused by modal differences between RGB and non-RGB images under cross-modal conditions. This network extracts features from RGB and non-RGB modal images separately through a parallel encoder, ensuring that image features from different modalities do not interfere with each other during encoding. Simultaneously, a contrastive learning strategy aligns the feature representations between modalities, mitigating the impact of cross-modal image imaging differences on feature consistency. In the depth estimation stage, RGB and non-RGB modal features are used to construct a cost volume, which is then combined with the Hourglass network and iterative optimization strategy to generate the final disparity prediction. This achieves a unified representation of cross-modal image features and improves the accuracy of the generated disparity map and the robustness of the cross-modal depth estimation model.

[0130] Figure 3 A schematic diagram illustrating the composition of a binocular vision depth estimation system based on cross-modal data, provided in this application embodiment, includes a monocular cross-modal data collection subsystem 10, a cross-modal data generation subsystem 20, a binocular matching subsystem 30, and a supervised optimization and result evaluation subsystem 40.

[0131] The monocular cross-modal data collection subsystem 10 is used to collect cross-modal monocular image data (such as RGB and infrared image pairs) and generate a registered cross-modal monocular image dataset through strict camera calibration and alignment.

[0132] The cross-modal data generation subsystem 20 includes a depth control module and a consistency constraint module, which generate cross-modal binocular image data using monocular cross-modal images and real depth information. The depth control module guides the depth consistency of the generated images, ensuring that the depth of the generated images matches the depth of the actual scene; the consistency constraint module enhances the geometric consistency between the left and right views of the generated images, optimizing the visual and depth matching of the generated images.

[0133] The binocular matching subsystem 30 is trained using the generated cross-modal binocular image data, learns the depth features of the cross-modal data through a deep control network, and optimizes the geometric relationship between the left and right views through a consistency constraint module to generate an accurate disparity map.

[0134] The supervised optimization and result evaluation subsystem 40 is used to construct a loss function to optimize the training of the aforementioned network, further save the trained network parameters, and generate the final disparity map. The generated disparity map is objectively evaluated using built-in evaluation metrics such as EPE and D1.

[0135] The connections between the above-mentioned components are as follows:

[0136] The output of the monocular cross-modal data collection subsystem is connected to the inputs of the cross-modal data generation subsystem and the binocular matching subsystem. The output of the cross-modal data generation subsystem is connected to the input of the binocular matching subsystem. Finally, the output of the binocular matching subsystem is connected to the input of the supervised optimization and result evaluation subsystem. The explanation of the binocular visual depth estimation method for complex weather environments in the foregoing embodiments also applies to the depth estimation system based on generated binocular images in this embodiment, and will not be repeated here.

[0137] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-modal binocular vision depth estimation method based on contrastive learning, characterized in that, Includes the following steps: Step 1: Collect monocular aligned cross-modal image data. Acquire RGB and non-RGB images through a multimodal camera system and perform rigorous camera calibration to achieve pixel-level alignment. Step 2: Construct a cross-modal binocular data generation model to generate standard-compliant binocular image data based on cross-modal image data aligned with monocular positioning, including depth-based disparity transformation and edge-aware inpainting modules; Step 3: Construct a cross-modal stereo depth estimation model, introduce feature pre-training based on contrastive learning and supervised constraint optimization, and train it using an aligned monocular cross-modal dataset and the generated cross-modal stereo dataset; Step 4: Save the training parameters, generate disparity images based on the input cross-modal binocular data, and complete inference and index evaluation.

2. The method as described in claim 1, characterized in that, The camera calibration in step 1 includes: intrinsic parameter calibration: obtaining the intrinsic parameter matrix of the camera, including focal length and principal point coordinates; extrinsic parameter calibration: obtaining the rotation matrix and translation vector between multimodal cameras; image alignment: mapping non-RGB images to the coordinate system of RGB images using projection formulas to achieve cross-modal image alignment.

3. The method as described in claim 1, characterized in that, The cross-modal binocular data generation model in step 2 includes: a depth-based disparity map generation module, which generates a disparity map using a monocular depth estimation model and adjusts the disparity range using a random scaling factor; and an edge-aware repair module, which repairs the hole regions caused by disparity mapping using edge mask detection and a stable diffusion model, preserving the edge information of objects.

4. The method as described in claim 3, characterized in that, The formula for generating the edge mask is: in, The horizontal gradient of the disparity map is represented by τ, which is the threshold used for edge detection.

5. The method as described in claim 1, characterized in that, The cross-modal binocular depth estimation model in step 3 includes: a parallel flow encoder architecture: extracting features independently for RGB and non-RGB modalities to generate multi-scale features; contrastive learning pre-training: aligning feature representations of different modalities by maximizing the cosine similarity of paired features; and supervised constraint optimization: optimizing the accuracy of disparity prediction through L1 loss and smoothing constraint loss.

6. The method as described in claim 5, characterized in that, The loss function for the contrastive learning is: in, is the cosine similarity matrix of the feature embedding; N2(·) denotes L2 normalization; G is the label matrix used to calculate the cross-entropy loss, with the elements on the diagonal being positive samples and the rest being negative samples; It is the cross-entropy loss function.

7. The method as described in claim 5, characterized in that, The loss function for the supervision constraint is: in, It is a true parallax map; λ is the gradient of the disparity map, used for smoothing constraints; λ is the weight of the smoothing loss.

8. The method as described in claim 1, characterized in that, The evaluation metrics in step 4 include: using EPE (End-Point Error) and D1 (Disparity One Pixel Error) to objectively evaluate the generated disparity map.

9. A cross-modal binocular vision depth estimation system based on contrastive learning, characterized in that, include: Monocular cross-modal data acquisition subsystem: used for acquiring and calibrating cross-modal monocular image data; Cross-modal data generation subsystem: Generates cross-modal binocular image data based on depth control and edge-aware restoration; Binocular matching subsystem: Trains a cross-modal binocular depth estimation model through contrastive learning and supervised constraints; Supervised optimization and result evaluation subsystem: Saves training parameters and generates disparity maps to complete performance evaluation.

Images