Method and system for generating and estimating disparity from any modality binocular stereo matching data
The cross-modal monocular image generation method using pixel alignment and block-level alignment solves the problems of cross-modal data scarcity and unstable stereo matching, and achieves high-precision disparity estimation in complex scenes. It is applicable to multiple modal combinations such as RGB, infrared, near-infrared, event, and polarization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2026-05-09
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies are prone to problems such as noise amplification, overexposure, or underexposure in RGB imaging under low light, strong dynamic range variation, weak texture, or complex reflection scenarios, resulting in unstable stereo matching. Furthermore, real cross-modal binocular data acquisition is difficult, the data scale is limited, and there is a lack of effective parallax supervision.
An arbitrary modal stereo matching method is adopted. Geometrically consistent cross-modal stereo training samples are generated by pixel-aligned cross-modal monocular image pairs. Disparity supervision is generated using a virtual stereo camera model and image completion technology. By combining block-level contrast representation alignment and disparity supervision, a multi-scale feature extraction and iterative optimization network is constructed.
It significantly alleviates the problem of cross-modal data scarcity, improves the stability and generalization ability of stereo matching in complex scenes, adapts to multiple modal combinations, and reduces the impact of local misalignment caused by imaging differences, occlusion, and noise.
Smart Images
Figure CN122434993A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, stereo vision, and multimodal learning technology, specifically to a method and system for generating arbitrary modal binocular stereo matching data and estimating disparity. Background Technology
[0002] Stereo matching is used to estimate pixel-level parallax between corrected left and right views and to reconstruct the 3D structure of a scene. It is a crucial foundation for tasks such as robot navigation, autonomous driving, augmented reality, and industrial inspection. Most existing learning-based stereo matching methods are designed for RGB-RGB same-modal inputs and can achieve high accuracy under normal lighting conditions.
[0003] In low-light, high dynamic range, weak texture, or complex reflection scenarios, RGB imaging is prone to noise amplification, overexposure, or underexposure, leading to unstable cost volume construction and increased parallax prediction errors. To improve imaging stability under extreme conditions, non-RGB sensors such as thermal infrared, near-infrared, event cameras, and polarization imaging have been introduced into 3D perception tasks. These modalities can provide complementary information such as thermal radiation, reflection, brightness variations, or polarization.
[0004] However, existing methods typically involve fusing non-RGB information into RGB before performing conventional stereo matching, or directly using a shared encoder structure within the same modality. Due to significant differences in the radiometric characteristics, texture representation, and local structure of cross-modal images, strict pixel-level alignment is difficult to achieve, especially in the presence of parallax displacement and occlusion, which can easily lead to local misalignment.
[0005] On the other hand, acquiring real cross-modal stereo data requires multi-sensor synchronization, stable extrinsic parameters, rigorous calibration, and geometric correction. The high complexity of hardware integration and the cost of data construction result in a limited scale of cross-modal stereo data available for supervised training. In contrast, pixel-aligned cross-modal monocular image pairs are easier to acquire, but such data typically lacks binocular disparity supervision that can be directly used for stereo matching training.
[0006] Therefore, a solution is needed that addresses both arbitrary modal stereo configurations and training data generation: on the one hand, it should be able to generate geometrically consistent stereo training samples with disparity supervision from pixel-aligned cross-modal monocular image pairs; on the other hand, it should provide stronger robustness to modal differences, local occlusion, and synthetic noise during cross-modal stereo matching training. Summary of the Invention
[0007] The purpose of this invention is to provide a method and system for generating stereo matching data and estimating disparity in arbitrary modal binoculars, which can solve the problems of scarcity of real cross-modal binocular data, inconsistent cross-modal feature distribution, and local pixel misalignment caused by disparity and occlusion.
[0008] To achieve the above objectives, the present invention adopts the following technical solution.
[0009] The arbitrary modal binoculars referred to in this specification mean that the left and right views of a binocular image pair are selected from the sets of RGB, infrared, near-infrared, event, and polarization modes, respectively; the preferred configuration is an RGB / non-RGB combination, and the modal positions of the left and right views can be interchanged according to the application scenario.
[0010] A method for generating and estimating disparity of arbitrary modal binocular stereo matching data includes the following steps.
[0011] Step 1: Obtain pixel-aligned cross-modal monocular image pairs.
[0012] The pixel-aligned cross-modal monocular image pairs can come from existing pixel-aligned datasets or be acquired by a multi-sensor imaging system. The multi-sensor imaging system includes an RGB camera and at least one non-RGB sensor, which includes, but is not limited to, infrared, near-infrared, event cameras, or polarized cameras. For self-acquired data, time synchronization, intrinsic parameter calibration, extrinsic parameter calibration, and image registration are used to achieve pixel-level alignment of images from different modalities in a unified coordinate system.
[0013] First, by calibrating the intrinsic parameters of the camera for each mode, the camera's intrinsic parameter matrix can be obtained:
[0014]
[0015] in,( , ) is the camera at ( ) and( Focal length in the direction of ) , () are the coordinates of the principal point of the image.
[0016] Secondly, by calibrating the extrinsic parameters of the multimodal camera, the rotation matrix between the two cameras can be obtained. Translation vector ):
[0017]
[0018] in, and These represent the coordinates of the RGB camera and the non-RGB camera in the world coordinate system, respectively. Rotation matrix. This describes the rotational relationship between the two cameras, and the translation vector. It describes the spatial displacement between the two cameras.
[0019] Finally, using the calibrated camera intrinsic and extrinsic parameters, non-RGB images can be projected onto the coordinate system of RGB images, achieving cross-modal image alignment. The projection formula is:
[0020]
[0021] in, These are 3D points from a non-RGB camera. These are the pixels projected onto the RGB image plane. It is the intrinsic parameter matrix of the RGB camera. This involves converting the extrinsic parameters of a non-RGB to RGB camera, ultimately resulting in a monocular aligned RGB image. and cross-modal images .
[0022] By following the steps above, we can ensure pixel-level alignment of cross-modal monocular images, providing high-quality registration data for subsequent cross-modal tasks.
[0023] Step 2: Construct a cross-modal stereo data generation engine to generate cross-modal stereo training samples with disparity supervision from pixel-aligned cross-modal monocular image pairs under a virtual stereo camera.
[0024] Specifically, a reference view in a pixel-aligned image pair, preferably an RGB image, is used as the source view and input into a monocular depth estimation model to obtain a relative depth or relative disparity representation. Then, under a virtual calibrated stereo camera model, the relative depth or relative disparity is mapped to a pixel-level disparity field to maintain the generated samples' clear geometric interpretability. The data generation engine outputs a synthesized right view and its corresponding disparity supervision label.
[0025] Step 2.1: Generate a pixel-level parallax field based on the virtual stereo camera parameters.
[0026] Stereo parallax is defined as the horizontal displacement difference between corresponding pixels in the left view and the right view, and its expression is:
[0027]
[0028] in, Represents pixels The disparity value, and These represent the horizontal coordinates of the corresponding pixels in the left and right views, respectively. This is for the relative disparity output from the monocular model. Through random scale parameters Convert it to pixel-level parallax Its expression is:
[0029]
[0030] in, From the preset range Internal random sampling is used to augment the baseline and disparity distribution in the training samples. In a preferred embodiment, the monocular depth estimation model employs DepthAnythingV2.
[0031] Step 2.2: Generate a virtual right view using forward mapping and image completion.
[0032] Based on the pixel-level disparity field, a forward mapping is performed on the source view to obtain a geometrically consistent initial draft of the virtual right view. Due to changes in viewpoint and occlusion, forward mapping often results in a hole area in the right view. Therefore, a visibility or occlusion mask needs to be constructed. ,in Represents pixels As can be seen in the mapping results, This indicates that the location belongs to an occluded or missing region. Then, an image completion model is used. The missing area is repaired to obtain the final right view, whose expression is:
[0033]
[0034] in, This represents the completed right view. The image completion model is preferably an image completion network or a diffusion completion model capable of reconstructing the missing region based on the surrounding visible context. Finally, the completed right view, the other modal image in the original pixel-aligned image pair, and the corresponding disparity supervision are combined to form a cross-modal binocular training sample. Preferably, a non-RGB image is used as the left view and the generated RGB image as the right view; in other embodiments, the configuration of the left and right modalities can be interchanged.
[0035] Step 3: Construct an arbitrary modality binocular stereo matching network, and jointly introduce block-level contrast representation alignment and disparity supervision during training.
[0036] Given an RGB image and a non-RGB image First, through modality-specific transformation Non-RGB images are converted into three-channel tensors to maintain consistency in the form of different modalities at the network input; subsequently, a feature extraction network with shared parameters is used. Extract multi-scale features from the left and right branches respectively.
[0037] Step 3.1: Multi-scale feature extraction and cost volume construction.
[0038] In a preferred embodiment, the feature extraction network uses EdgeNeXt-S as its backbone, downsampling the input to the original resolution. and in The scale introduces DepthAnythingV2 features, which are then recovered through an upsampling module with skip connections. , , and The expression for the multi-scale feature with four scales is as follows:
[0039]
[0040] in, Indicates the feature scale; It is a multi-scale feature of the RGB modality. These are non-RGB modal multi-scale features. The multi-scale features are used in... A three-dimensional cost volume is constructed at the scale to provide a foundation for subsequent disparity prediction.
[0041] Step 3.2: Initial disparity prediction and iterative optimization.
[0042] Using the aforementioned cost body, via the Hourglass network An initial disparity prediction is generated, and then ConvGRU is used for iterative refinement to obtain the final disparity result, which is expressed as follows:
[0043]
[0044]
[0045] in, Indicates the initial disparity prediction. Indicates the first The disparity prediction results after the next iteration.
[0046] Step 3.3: Introduce block-level contrastive representation alignment in stereo matching training.
[0047] Because cross-modal imaging differences, parallax shift, occlusion, and synthetic noise can cause local pixel-level deviations in corresponding regions, this invention does not employ strict pixel-level alignment. Instead, it performs spatial downsampling on the feature map and divides it into... Each feature block is a non-overlapping feature block. The block size or number of blocks is adaptively set based on the estimated alignment error: when the error is large, larger blocks are used to absorb local misalignment, and when the error is small, smaller blocks are used to preserve fine-grained geometric constraints.
[0048] Feature blocks at the same spatial location as the non-RGB branch and the RGB branch are taken as positive samples, and feature blocks at other locations are taken as negative samples. A symmetric cross-entropy contrastive loss is constructed based on cosine similarity. Its expression is:
[0049]
[0050]
[0051] in, Represents block-level embedding features of non-RGB branches. Represents the block-level embedding features of the RGB branch. Indicates L2 normalization, Represents the label matrix, It is the cross-entropy loss function. Block-level contrastive representation alignment directly affects the stereo matching training process to improve consistency across modal feature spaces.
[0052] Step 3.4: Optimize network parameters jointly using parallax supervision and block-level contrast loss.
[0053] In effective parallax mask Below, L1 loss is used as the disparity supervision loss. Its expression is:
[0054]
[0055] in, It is a true parallax map. It is a predicted disparity map.
[0056] In one implementation, the total loss is a weighted sum of contrast loss and disparity supervision loss:
[0057]
[0058] in, and This represents the loss weight. In a preferred embodiment, the model is first pre-trained using the cross-modal stereo samples generated in step 2, and then fine-tuned on real cross-modal stereo data as needed.
[0059] Step 4: Save the trained network parameters and perform inference and evaluation on any input modal stereo image pair.
[0060] During the inference phase, the corrected arbitrary modal binocular image pairs are input into the model, and a scene disparity map is output. When the camera focal length and baseline are known, the disparity map can be further converted into a depth map. Preferred model evaluation metrics include endpoint error (EPE), D1 index, and an error rate greater than 2 pixels.
[0061] Based on the above method, the present invention also provides an arbitrary modal binocular stereo matching data generation and disparity estimation system, including a pixel alignment data acquisition and registration subsystem 10, a cross-modal binocular data generation subsystem 20, an arbitrary modal binocular stereo matching subsystem 30, and a training and inference evaluation subsystem 40.
[0062] The pixel alignment data acquisition and registration subsystem 10 is used to acquire, import, or construct pixel-aligned cross-modal monocular image pairs. The cross-modal binocular data generation subsystem 20 includes a monocular depth estimation unit, a disparity mapping unit, a forward mapping unit, an occlusion mask generation unit, and an image completion unit, used to generate cross-modal binocular training samples with disparity supervision. The arbitrary modal binocular stereo matching subsystem 30 includes a modality-specific transformation unit, a feature extraction unit, a block-level contrast representation alignment unit, a cost volume construction and initial disparity prediction unit, and an iterative optimization unit, used to output disparity prediction results. The training and inference evaluation subsystem 40 is used to jointly train the above units, save parameters, and complete inference and accuracy evaluation.
[0063] The connection relationships between the modules are as follows: the output terminal of subsystem 10 is connected to the input terminal of subsystem 20, the output terminal of subsystem 20 is connected to the training input terminal of subsystem 30, and the output terminal of subsystem 30 is connected to the input terminal of subsystem 40.
[0064] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method steps of any of the above embodiments.
[0065] Beneficial effects
[0066] Compared with the prior art, the present invention has at least the following beneficial effects.
[0067] 1. This invention can expand more readily available pixel-aligned cross-modal monocular image pairs into geometrically consistent cross-modal binocular training samples with disparity supervision, significantly alleviating the problem of scarcity of real cross-modal binocular data.
[0068] 2. This invention is designed for arbitrary modal binocular configurations, and can adapt to multiple modal combinations such as RGB, infrared, near-infrared, event, and polarization, and allows the left and right view modalities to be interchanged according to the application scenario.
[0069] 3. The present invention adopts a block-level rather than pixel-level contrast characterization alignment method, which provides the necessary spatial tolerance for cross-modal correspondence and can reduce the local misalignment effects caused by differences in imaging mechanisms, parallax displacement, occlusion and synthetic noise.
[0070] 4. This invention utilizes a combination of multi-scale cost volume, Hourglass initial disparity prediction, and ConvGRU iterative optimization to improve the stability and generalization ability of cross-modal stereo matching under low illumination, weak texture, and complex lighting conditions. Attached Figure Description
[0071] Figure 1 This is a flowchart of the arbitrary modal binocular stereo matching data generation and disparity estimation method of the present invention.
[0072] Figure 2 This is a schematic diagram of the data generation engine and arbitrary modal stereo matching network structure of the present invention.
[0073] Figure 3 This is a schematic diagram of the system composition and connection relationship of the present invention. Detailed Implementation
[0074] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0075] Example 1
[0076] like Figure 1 As shown, a method for generating and estimating disparity of arbitrary modal binocular stereo matching data includes the following steps.
[0077] Step 1: Obtain pixel-aligned cross-modal monocular image pairs.
[0078] Step 1.1: Obtain pixel-aligned cross-modal monocular image pairs. These image pairs can be directly derived from existing pixel-aligned datasets or acquired automatically by the multi-sensor imaging system. In one specific embodiment, one or more data sources from FANVID, LLVIP, EventScape, and SPW can be used to construct cross-modal image pairs such as RGB-near-infrared, RGB-thermal infrared, RGB-event, and RGB-polarization, respectively. To adapt to arbitrary modal binocular configurations, the modalities of the left and right views are not limited to fixed positions but can be any two selected from the RGB, infrared, near-infrared, event, and polarization modal sets for combination.
[0079] Step 1.2: When using a self-built multi-sensor system to acquire data, first perform time synchronization, intrinsic parameter calibration, extrinsic parameter calibration, and image registration for each modal sensor to ensure pixel-level alignment of images from different modalities in a unified coordinate system. The camera intrinsic parameter matrix can be represented as:
[0080]
[0081] in, and These represent the camera's focal length in the horizontal and vertical directions, respectively. and These represent the coordinates of the principal point of the image.
[0082] Step 1.3: Further, the rotation matrix and translation vector between the RGB camera and the non-RGB camera are obtained through extrinsic parameter calibration. Their relationship can be expressed as:
[0083]
[0084] in, Represents a 3D point in the RGB camera coordinate system. Represents a 3D point in a non-RGB camera coordinate system. Let be a rotation matrix. It is a translation vector.
[0085] Step 1.4: Project the 3D points in the non-RGB image onto the RGB image coordinate system to complete cross-modal image registration. The projection relationship can be expressed as:
[0086]
[0087] in, Represents 3D points from a non-RGB camera. This represents the projected pixel coordinates. Through the above processing, pixel-aligned cross-modal monocular image pairs are obtained, providing an input basis for subsequent binocular data generation and disparity estimation.
[0088] Step 2: Build a cross-modal stereo data generation engine to generate cross-modal stereo training samples with disparity supervision under a virtual stereo camera model.
[0089] Step 2.1: Select one image from the pixel-aligned image pair obtained in Step 1 as a reference view. In a preferred embodiment, an RGB image is used as the source view for generating the virtual right view, and another modal image is used as a candidate left view. Subsequently, the RGB image is input into the monocular depth estimation model DepthAnythingV2 to obtain a relative depth or relative disparity representation.
[0090] Step 2.2: Under the virtual calibrated stereo camera, map the depth or relative parallax to a pixel-level parallax field. For a known virtual focal length... and baseline In this case, depth and parallax satisfy the following relationship:
[0091]
[0092] in, For parallax, For depth.
[0093] Step 2.3: In stereo matching, disparity is defined as the horizontal displacement difference between corresponding pixels in the left and right views, and its expression is:
[0094]
[0095] in, Represents pixels The disparity value, and These represent the pixels in the left view. Pixels corresponding to the right view The horizontal coordinates.
[0096] Step 2.4: When the monocular model outputs a normalized relative disparity map At that time, a random scaling parameter is used to convert it into a pixel-level disparity map. The transformation relationship can be expressed as:
[0097]
[0098] in, This is a scale factor obtained by random sampling within a preset interval. In a preferred embodiment, , This is to expand the baseline and disparity distributions that may appear in the training samples.
[0099] Step 2.5: Based on the pixel-level parallax field, perform forward mapping on the source view to obtain a geometrically consistent initial draft of the virtual right view. Because there are occluded and invisible areas in the new viewpoint, it is necessary to generate either a visibility mask or an occlusion mask simultaneously. Its definition can be written as:
[0100]
[0101] Step 2.6: Use the image completion model to... Repair the missing areas to obtain the final right view. The repair process can be represented as follows:
[0102]
[0103] in, The image completion model is represented as follows: In a preferred embodiment, the image completion model may be a diffusion completion model or other image completion networks that can recover the occluded area based on the surrounding context.
[0104] Step 2.7: Combine the other modal image retained in Step 1, the right view obtained in Step 2.6, and the corresponding disparity supervision label to form a cross-modal binocular training sample. In one specific embodiment, the non-RGB image can be used as the left view, and the generated RGB image can be used as the right view, thereby forming a training pair with pixel-level disparity supervision; in other embodiments, the configuration of the left and right modalities can also be interchanged according to the application scenario.
[0105] Step 3: Construct an arbitrary modality binocular stereo matching network, and jointly introduce block-level contrast representation alignment and disparity supervision during training.
[0106] Step 3.1: Convert the input non-RGB image into a three-channel tensor using a modality-specific transform to maintain a data interface format consistent with RGB images. Its input format can be represented as:
[0107]
[0108] in, Indicates a non-RGB left view. This indicates a modality-specific transformation.
[0109] Step 3.2: Convert the RGB image from the right view. The transformed non-RGB image and the image itself are fed into a feature extraction network with shared parameters. Preferably, the feature extraction network uses EdgeNeXt-S as its backbone, and downsamples the input to its original resolution. and in The scale-fusing DepthAnythingV2 features are then used to recover multi-scale features via an upsampling module with skip connections. Its expression can be represented as:
[0110]
[0111] in, Indicates the feature scale; It is a multi-scale feature of the RGB modality. It is a multi-scale feature of the RGB modality.
[0112] Step 3.3: Construct a 3D cost volume using the above multi-scale features, and then apply it through an Hourglass network. An initial disparity prediction is generated, and then iteratively refined using the ConvGRU module. The relationship between the initial prediction and the iterative update can be expressed as follows:
[0113]
[0114]
[0115] in, This represents the initial disparity prediction result. Indicates the first The disparity prediction results after the next iteration.
[0116] Step 3.4: Considering the differences in cross-modal imaging mechanisms, parallax displacement, occlusion, and potential local noise in the data generated in Step 2, which could lead to pixel-level misalignment in the corresponding regions, this embodiment does not employ a strict pixel-level alignment method. Instead, it performs spatial downsampling on the feature map and divides it into several non-overlapping feature blocks. The block size is adaptively set based on the estimated alignment error: when the error is large, a larger block is used to absorb local misalignment; when the error is small, a smaller block is used to preserve fine-grained geometric constraints.
[0117] Step 3.5: Use feature blocks in the same spatial location as the RGB branch as positive samples, and feature blocks in other locations as negative samples to construct block-level contrastive representation learning. Let the block-level embedding sets of the left and right branches be respectively... , The cosine similarity matrix can then be expressed as:
[0118]
[0119] Corresponding symmetric cross-entropy contrast loss It can be represented as:
[0120]
[0121] in, express Normalization Represents the label matrix, It is the cross-entropy loss function. Feature blocks at the same spatial location are considered positive samples, and the remaining feature blocks are considered negative samples.
[0122] Step 3.6: Under the effective disparity mask, introduce the disparity prediction results... Supervision is required to ensure the network learns accurate pixel-level matching relationships. Parallax supervision loss. It can be represented as:
[0123]
[0124] in, Indicates the effective parallax mask. Indicates true parallax label, This represents the network's predicted parallax.
[0125] Step 3.7: Weight the block-level contrast loss and disparity supervision loss, and jointly optimize the entire network parameters. The total loss can be expressed as:
[0126]
[0127] in, and The loss weights are used. In a preferred training strategy, the model is first pre-trained using the cross-modal stereo samples generated in step 2, and then fine-tuned on real cross-modal stereo data as needed to further improve the model's adaptability under real imaging conditions.
[0128] Step 4: Save the trained network parameters and perform inference and evaluation on any input modal stereo image pair.
[0129] Step 4.1: After training, save the network parameters. When disparity estimation is needed, input any modal binocular image after epipolar correction into the model. The model outputs the disparity prediction result of the scene, and its inference form can be expressed as:
[0130]
[0131] in, The parameter is Arbitrary modal binocular stereo matching model.
[0132] Step 4.2: When the camera focal length is known and baseline At that time, it can be determined from the parallax results. Further recovery of the depth map can be represented as follows:
[0133]
[0134] in, This represents the depth value.
[0135] Step 4.3: When evaluating model performance, endpoint error (EPE), D1 index, and error rate greater than 2 pixels can be used to quantitatively analyze the accuracy and robustness of the disparity map. For scenarios with low illumination, weak texture, strong dynamic range variations, and complex reflections, this embodiment improves the stability and generalization ability of cross-modal stereo matching through a joint design of arbitrary modal binocular configuration, synthetic data supervision, and block-level contrast characterization alignment.
[0136] In summary, this embodiment generates geometrically consistent and disparity-supervised arbitrary modal binocular training samples by pixel-aligned cross-modal monocular image pairs. Furthermore, it introduces block-level contrast representation alignment and disparity supervision for joint optimization in stereo matching training. This approach can alleviate the scarcity of real cross-modal binocular data and the inconsistency in cross-modal feature distribution, thereby improving the accuracy and robustness of disparity estimation in complex scenarios.
[0137] Figure 2 The structure shown consists of two parts: a data generation engine and an arbitrary modal stereo matching network. The data generation engine is responsible for generating cross-modal binocular training samples from pixel-aligned cross-modal monocular image pairs; the arbitrary modal stereo matching network obtains high-precision disparity through modality transformation, multi-scale feature extraction, block-level contrast alignment, cost volume construction, and iterative optimization.
[0138] Figure 3 In the system shown, the pixel alignment data acquisition and registration subsystem 10 is responsible for data preparation; the cross-modal binocular data generation subsystem 20 is responsible for training sample generation; the arbitrary modal binocular stereo matching subsystem 30 is responsible for network training and inference; and the training and inference evaluation subsystem 40 is responsible for parameter saving, depth / disparity output, and accuracy evaluation.
[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any equivalent substitutions, improvements, and modifications made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for generating and estimating disparity of arbitrary modal binocular stereo matching data, characterized in that, The process includes the following steps: S1, acquiring pixel-aligned cross-modal monocular image pairs; S2, constructing a cross-modal stereo data generation engine under a virtual stereo camera based on the pixel-aligned cross-modal monocular image pairs, generating disparity-supervised cross-modal stereo training samples using monocular depth estimation, pixel-level disparity mapping, forward mapping, occlusion region recognition, and image completion; S3, constructing an arbitrary modal stereo matching network, performing modality-specific transformations, multi-scale feature extraction, block-level contrast representation alignment, cost volume construction, initial disparity prediction, and iterative optimization on the input stereo image pairs to obtain a predicted disparity map; S4, using the disparity-supervised cross-modal stereo training samples, and jointly training the arbitrary modal stereo matching network by combining block-level contrast loss and disparity supervision loss; S5, inputting the arbitrary modal stereo image pairs to be inferred into the trained arbitrary modal stereo matching network, outputting a disparity map, and converting the disparity map into a depth map when the camera focal length and baseline are known.
2. The method as described in claim 1, characterized in that, The left and right views of the arbitrary modal binocular image pair are respectively selected from RGB, infrared, near-infrared, event, and polarization modes, and at least one side is different from the other side; the pixel-aligned cross-modal monocular image pair comes from a pixel-aligned dataset or data obtained through multi-sensor calibration, synchronization, and registration.
3. The method as described in claim 1, characterized in that, In step S2, the reference view in the pixel-aligned cross-modal monocular image pair is used as the source view. The monocular depth estimation model is then used to obtain a relative depth or relative disparity representation, and this representation is applied under a virtual calibrated stereo camera model using random scale parameters. Map the relative depth or relative disparity to a pixel-level disparity field. ,in .
4. The method as described in claim 3, characterized in that, Step S2 further includes: performing forward mapping on the source view based on the pixel-level parallax field to generate a geometrically consistent virtual right view draft. ; Construct a visibility or occlusion mask M, where This indicates that pixel p is visible in the mapping result. This indicates that pixel p is located in the occluded / missing region; an image completion model is used. The missing areas are filled in to obtain the final right view. ,in .
5. The method as described in claim 1, characterized in that, In step S3, a mode-specific transformation is first performed on the non-RGB input. It is converted into a three-channel tensor, and then processed by a feature extraction network with shared parameters. Multi-scale features are extracted to obtain the reference view feature set and the target view feature set.
6. The method as described in claim 5, characterized in that, The feature extraction network Using EdgeNeXt-S as the feature encoding backbone, the input features are downsampled to 1 / 32 of the original resolution, and after introducing DepthAnythingV2 features at the 1 / 4 scale, the multi-scale features at four scales of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 are recovered through an upsampling module with skip connections. A cost volume is constructed at a 1 / 4 scale, and initial disparity predictions are generated using an Hourglass network. The final disparity prediction is then obtained through ConvGRU iterative optimization. .
7. The method as described in claim 1, characterized in that, Step S3, block-level contrastive representation alignment, includes: spatial downsampling of the multi-scale feature map and dividing it into... The feature blocks are non-overlapping; cross-modal feature blocks at the same spatial location are used as positive samples, and feature blocks at other locations are used as negative samples; the block size is adaptively set according to the estimated alignment error or the quality of the training data, increasing the block size when the alignment error increases and decreasing the block size when the alignment error decreases.
8. The method as described in claim 7, characterized in that, The block-level comparison loss employs a symmetric cross-entropy loss based on cosine similarity, expressed as: ; ;in, Represents block-level embedding features of non-RGB branches. Represents the block-level embedding features of the RGB branch. Indicates L2 normalization, The label matrix is represented; the disparity supervision loss uses an effective pixel mask. The L1 loss is expressed as follows: .
9. A system for generating stereo matching data and estimating disparity in arbitrary modal binoculars, characterized in that, include: The system comprises a pixel-aligned data acquisition and registration module, a cross-modal binocular data generation module, an arbitrary modal binocular stereo matching module, and a training and inference evaluation module. Specifically, the pixel-aligned data acquisition and registration module acquires pixel-aligned cross-modal monocular image pairs; the cross-modal binocular data generation module generates cross-modal binocular training samples with disparity supervision; the arbitrary modal binocular stereo matching module performs modality-specific transformations, multi-scale feature extraction, block-level contrast representation alignment, cost volume construction, initial disparity prediction, and iterative optimization; and the training and inference evaluation module jointly trains the above modules, saves parameters, and performs inference and accuracy evaluation. The output of the pixel-aligned data acquisition and registration module is connected to the inputs of the cross-modal binocular data generation module and the arbitrary modal binocular stereo matching module, respectively. The output of the cross-modal binocular data generation module is connected to the training input of the arbitrary modal binocular stereo matching module, and the output of the arbitrary modal binocular stereo matching module is connected to the input of the training and inference evaluation module.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.