A cross-spectrum stereo depth estimation method, device, equipment and medium

Abstract

This invention relates to a method, apparatus, device, and medium for cross-spectral stereo depth estimation. The method includes: inputting acquired left-eye thermal infrared and right-eye visible light images into a cross-spectral depth estimation network to obtain a disparity map. The cross-spectral depth estimation network includes: an encoder module for extracting features from the left-eye thermal infrared and right-eye visible light images respectively, obtaining thermal infrared features and visible light features; a guided fusion module for generating a pixel-level reliability weight map based on the right-eye visible light image, and fusing the visible light and thermal infrared features using the reliability weight map to obtain cross-modal fused features; and a decoder module for separating high and low frequencies in the cross-modal fused features, enhancing and stitching the obtained high-frequency and low-frequency features respectively, and obtaining a disparity map based on the stitched enhanced features. This invention can improve the robustness and accuracy of all-weather depth estimation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for cross-spectral stereo depth estimation. Background Technology

[0002] With the development of autonomous driving, drone navigation, security monitoring, and other fields, accurate depth perception systems have become a core component. However, existing depth estimation methods typically rely on visible light images and are easily affected by lighting conditions. In low-light or complex lighting environments, traditional visible light imaging methods perform poorly, leading to a significant decrease in depth estimation accuracy and failing to meet all-weather requirements. Therefore, in recent years, transspectral technologies, especially thermal infrared imaging (TIR), have begun to be introduced into depth estimation systems to provide stable structural information under low-light and nighttime conditions.

[0003] Existing transspectral depth estimation techniques can be broadly categorized as follows: (1) Supervised transspectral depth estimation, which regresses depth by fusing VIS and TIR information, but relies on high-cost pixel-level depth ground truth; (2) Unsupervised / self-supervised methods, which utilize stereo geometric reprojection loss for training, but the appearance differences of transspectral images are large, and photometric consistency is difficult to establish directly. To address this issue, researchers have proposed several solutions, among which cross-domain alignment methods based on image transfer or feature transfer are the mainstream approach. These methods first map TIR to the VIS domain or align features to the VIS feature space before performing depth decoding. However, the training process is complex and prone to introducing pseudo-textures or instability. There are also methods that directly perform cross-modal feature fusion, but they lack adaptive evaluation of the degree of visible light degradation and are prone to negative transfer under noisy VIS input at night. In addition, in terms of decoder design, cross-spectral inputs often contain both low-frequency structural information (more from TIR) and high-frequency texture information (more from VIS under good lighting). Traditional convolutional decoders usually mix and process different frequency information, and edge and texture details are easily lost during upsampling, resulting in blurred boundaries or texture artifacts. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a cross-spectral stereo depth estimation method, apparatus, device and medium that can improve the robustness and accuracy of all-weather depth estimation.

[0005] The technical solution adopted by this invention to solve its technical problem is: to provide a transspectral stereo depth estimation method, comprising the following steps:

[0006] Acquire thermal infrared images from the left eye and visible light images from the right eye;

[0007] The left-eye thermal infrared image and the right-eye visible light image are input into a transspectral depth estimation network to obtain a disparity map; wherein, the transspectral depth estimation network includes:

[0008] The encoder module is used to extract features from the left-eye thermal infrared image and the right-eye visible light image respectively to obtain thermal infrared features and visible light features;

[0009] The guided fusion module is used to generate a pixel-level reliability weight map based on the right eye visible light image, and to fuse the visible light features and thermal infrared features using the reliability weight map to obtain cross-modal fusion features;

[0010] The decoder module is used to separate the high-frequency and low-frequency features of the cross-modal fusion, enhance and stitch the obtained high-frequency and low-frequency features respectively, and obtain a disparity map based on the stitched enhanced features.

[0011] The encoder module includes:

[0012] A visible light encoder is used to extract features from the right eye's visible light image to obtain visible light features;

[0013] An infrared optical encoder is used to extract features from the left eye thermal infrared image to obtain thermal infrared features.

[0014] The parameters of the visible light encoder are obtained based on training of a self-supervised visible light basic network. The self-supervised visible light basic network adopts an encoder-decoder architecture, with visible light stereo pairs as inputs and left and right parallax as outputs.

[0015] The loss function of the self-supervised visible light basic network is expressed as:

[0016] ;

[0017] in, For the loss of self-supervised visible light basic networks, and Both are losses of photometric uniformity, expressed as: , Number of pixels and The positions in the left and right views are respectively pixel values, To obtain the position in the left view by reprojecting the right view using the left disparity predicted by a self-supervised visible light fundamental network, the position is... pixel values, To obtain the position in the right view by reprojecting the right disparity predicted by the left view using a self-supervised visible light fundamental network, the position is... pixel values, For similarity calculation, These are the weighting coefficients; and Both are edge-aware smoothing losses, expressed as: ,in, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by a self-supervised visible light-based network, respectively. and These are the left view and the right view, respectively. and Both are left-right consistency losses, expressed as: ,in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map obtained by reprojecting onto the left view coordinate system; The weights for edge-aware smoothing loss are... The weights for the left-right consistency loss.

[0018] The guided fusion module includes:

[0019] The depth estimation section is used to perform depth estimation on the right eye visible light image to obtain the prediction variance of the visible light pixels;

[0020] The reliability attention generation section is used to generate a pixel-level reliability weight map based on the prediction variance of the visible light pixels.

[0021] The fusion section is used to guide the visible light features using the pixel-level reliability weight map, and then fuse the guided visible light features with the thermal infrared features to obtain cross-modal fused features.

[0022] The decoder module includes several decoding layers, and each decoding layer includes:

[0023] The high- and low-frequency feature separation section is used to perform average pooling on the input features to obtain low-frequency components, then upsample the low-frequency components, and subtract the upsampled low-frequency components from the input features to obtain high-frequency components.

[0024] The low-frequency enhancement part is used to enhance the nonlocal structural information of the low-frequency components using a lightweight self-attention approximation mechanism to obtain enhanced low-frequency features;

[0025] The high-frequency enhancement part is used to extract contextual information of different receptive fields in parallel from the high-frequency components using multi-scale dilated convolution branches, so as to obtain enhanced high-frequency features.

[0026] The splicing and fusion section is used to splice the enhanced low-frequency features and enhanced high-frequency features, which are then weighted by convolutional layers and channel attention, and fused with the input features through residual connections before being output to the next level of decoding layer.

[0027] The loss function of the transspectral depth estimation network is expressed as:

[0028] ;

[0029] in, The loss function for the cross-spectral depth estimation network, The loss across spectral stages is expressed as: , The network reconstruction loss is represented as: ,in, This is the thermal infrared image of the left eye. This is the visible light image of the right eye. It has thermal infrared characteristics. It has visible light characteristics. For infrared reconstruction network, Reconstructing the network for visible light; and The loss for smoothing across spectral edges is expressed as: ,in, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by the transspectral depth estimation network, respectively. and The cross-spectral left-right consistency loss is expressed as: ,in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map obtained by reprojecting it onto the left view coordinate system. The weights for the cross-spectral edge smoothing loss, The weights for cross-spectral left-right consistency loss; The cross-spectral depth cyclic consistency loss is expressed as: , and The image pairs are respectively Input the left and right disparities obtained from the transspectral depth estimation network. This is a left-eye visible light image constructed from the right-eye visible light image and the left disparity predicted by a transspectral depth estimation network. The thermal infrared image of the right eye is constructed based on the left eye thermal infrared image and the right disparity predicted by the transspectral depth estimation network. The weights for cross-spectral depth cyclic consistency loss.

[0030] The technical solution adopted by this invention to solve its technical problem is: to provide a transspectral stereo depth estimation device, comprising:

[0031] The acquisition module is used to acquire the left eye thermal infrared image and the right eye visible light image;

[0032] An estimation module is used to input the left-eye thermal infrared image and the right-eye visible light image into a transspectral depth estimation network to obtain a disparity map; wherein, the transspectral depth estimation network includes:

[0033] The encoder module is used to extract features from the left-eye thermal infrared image and the right-eye visible light image respectively to obtain thermal infrared features and visible light features;

[0034] The guided fusion module is used to generate a pixel-level reliability weight map based on the right eye visible light image, and to fuse the visible light features and thermal infrared features using the reliability weight map to obtain cross-modal fusion features;

[0035] The decoder module is used to separate the high-frequency and low-frequency features of the cross-modal fusion, enhance and stitch the obtained high-frequency and low-frequency features respectively, and obtain a disparity map based on the stitched enhanced features.

[0036] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned transspectral stereo depth estimation method.

[0037] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned transspectral stereo depth estimation method.

[0038] Beneficial effects

[0039] By adopting the above-mentioned technical solutions, this invention has the following advantages and positive effects compared with the prior art: This invention uses a pixel-level reliability weight map to guide cross-spectral feature fusion, which can adaptively identify and suppress noise interference in visible light degradation areas, fully preserve visible light texture details in well-lit areas, improve the robustness of cross-modal feature fusion, and avoid negative transfer problems; the decoder in this invention explicitly separates and specifically enhances the high and low frequencies of the fused features, effectively improving the global structure recovery capability dominated by thermal infrared and the edge texture detail restoration capability dominated by visible light, solving the problems of high and low frequency information mixing and detail loss in traditional decoding; the modular design of the cross-spectral depth estimation network in this invention is compatible with binocular cross-spectral imaging systems, and the inference process does not rely on the true depth value, fully utilizing the illumination robustness of thermal infrared and the texture richness of visible light, significantly improving the accuracy and stability of depth estimation in all-weather scenarios, and adapting to practical application needs such as autonomous driving and drone navigation. Attached Figure Description

[0040] Figure 1 This is a framework diagram of the cross-spectral depth estimation network in the first embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the guiding fusion module in the first embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the decoder in the first embodiment of the present invention;

[0043] Figure 4 These are schematic diagrams illustrating the parallax visualization results in different scenarios. Detailed Implementation

[0044] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0045] The first embodiment of this invention relates to a transspectral stereo depth estimation method. This method is designed for a geometrically calibrated transspectral stereo imaging system. Without relying on the true depth value, it achieves noise-resistant fusion through a visible light reliability-guided transspectral feature fusion module (VRGF), and introduces a high-low frequency feature adaptive enhancement module (HLFAE) during the decoding stage to improve structure and detail recovery capabilities, thereby enhancing the robustness and accuracy of all-weather depth estimation. The transspectral stereo depth estimation method of this embodiment specifically includes the following steps:

[0046] Step 1: Acquire the thermal infrared image of the left eye and the visible light image of the right eye;

[0047] Step 2: Input the left-eye thermal infrared image and the right-eye visible light image into the transspectral depth estimation network to obtain a disparity map.

[0048] like Figure 1 As shown, the transspectral depth estimation network includes:

[0049] The encoder module is used to extract features from the left-eye thermal infrared image and the right-eye visible light image respectively to obtain thermal infrared features and visible light features;

[0050] The guided fusion module is used to generate a pixel-level reliability weight map based on the right eye visible light image, and to fuse the visible light features and thermal infrared features using the reliability weight map to obtain cross-modal fusion features;

[0051] The decoder module is used to separate the high-frequency and low-frequency features of the cross-modal fusion, enhance and stitch the obtained high-frequency and low-frequency features respectively, and obtain a disparity map based on the stitched enhanced features.

[0052] The encoder module in this embodiment includes:

[0053] A visible light encoder is used to extract features from the right eye's visible light image to obtain visible light features;

[0054] An infrared optical encoder is used to extract features from the left eye thermal infrared image to obtain thermal infrared features.

[0055] The parameters of the visible light encoder are obtained based on a self-supervised visible light fundamental network (VLMN), which aims to learn the geometric structural features in visible light images, providing geometric priors for subsequent cross-spectral fusion. This self-supervised VLMN can employ an encoder-decoder architecture, with its input being visible light stereo pairs. Visible light encoder with shared weights Extracting multi-scale features, i.e. ,in, and These are the left-eye multi-scale features and right-eye multi-scale features extracted by the visible light encoder, respectively, and then processed by the decoder. Output the left and right parallax, i.e.: ,in, and These are the left and right disparities predicted by the self-supervised visible light base network, respectively. The visible light encoder in this embodiment... and decoder U-Net-style skip connections are used to fuse multi-scale features.

[0056] The loss function of this self-supervised visible light basic network includes photometric consistency loss, edge-aware smoothing loss, and left-right consistency loss.

[0057] This implementation method utilizes the predicted left parallax to reproject the right view, resulting in the left view. The right view is obtained by reprojecting the predicted right parallax onto the left view. ,Right now: , .

[0058] The loss of photometric uniformity can be expressed as:

[0059] ;

[0060] in, Number of pixels and The positions in the left and right views are respectively pixel values, To obtain the position in the left view by reprojecting the right view using the left disparity predicted by a self-supervised visible light fundamental network, the position is... pixel values, To obtain the position in the right view by reprojecting the right disparity predicted by the left view using a self-supervised visible light fundamental network, the position is... pixel values, For similarity calculation, These are the weighting coefficients.

[0061] Edge-aware smoothing loss can be expressed as:

[0062] ;

[0063] in, and These represent the gradients of the image in the x and y directions, respectively.

[0064] The left-right consistency loss can be expressed as:

[0065] ;

[0066] The loss function of this self-supervised visible light-based network is:

[0067] ;

[0068] in, The weights for edge-aware smoothing loss are... The weights for the left-right consistency loss.

[0069] During the training phase of the cross-spectral depth estimation network, to avoid cross-modal loss interfering with the visible light geometric representation, the visible light encoder can be modified. Freeze the parameters, or freeze the visible light encoder. The output features are detached from the gradient, so that they serve only as a stable geometric reference branch and do not receive inverse gradients across spectral branches.

[0070] To address the issues of unreliable visible light images and the potential for noise introduction during direct fusion in low-light, overexposed, or weak-texture scenes, this implementation's guided fusion module adaptively generates a reliability weight map based on learned pixel-level uncertainty. This map is then used to spatially weight visible light features before fusing them with thermal infrared features. For example... Figure 2 As shown, the guided fusion module includes: a depth estimation part, used to perform depth estimation on the right eye visible light image to obtain the prediction variance of visible light pixels; a reliability attention generation part, used to generate a pixel-level reliability weight map based on the prediction variance of the visible light pixels; and a fusion part, used to guide the visible light features using the pixel-level reliability weight map and fuse the guided visible light features with thermal infrared features to obtain cross-modal fusion features.

[0071] Specifically, this implementation uses a D-Net model to perform depth estimation on the right-eye visible light image, and estimates the pixel depth of the right-eye visible light image. The depth estimation model is based on a Gaussian distribution, i.e.:

[0072]

[0073] in, and Pixels The predicted mean and predicted variance. A larger variance indicates that the depth estimate at that location is less reliable, and the visible light features at that pixel are less reliable (e.g., dark, overexposed, or noisy areas), and their impact needs to be reduced during fusion.

[0074] A reliability attention weight map can be generated based on the prediction variance of each visible light pixel. Its variation with prediction variance Monotonically increasing (with larger weights for unreliable regions), for example, can be achieved using a normalized mapping, i.e.:

[0075] ;

[0076] Visible light characteristics and thermal infrared characteristics are respectively and Then, a reliability weighting map is used to guide the visible light features, and the guided visible light features are fused with the thermal infrared features, which is expressed as:

[0077] ;

[0078] Among them, when the variance When the value is larger (low confidence), The smaller the variance, the more it suppresses visible light characteristics. The smaller the confidence level (high confidence), the more visible light texture details are preserved, thus suppressing low-confidence areas and preserving high-confidence areas.

[0079] Through the above fusion, in well-lit areas, the network can utilize the high-frequency texture details of visible light to improve the depth boundary and detail quality; in poorly lit or visible light degraded areas, the weights are automatically reduced, causing the network to revert to being dominated by thermal infrared structural information, thereby improving the robustness of cross-spectral fusion.

[0080] To simultaneously improve the global structure recovery capability dominated by thermal infrared and the edge / texture detail quality after cross-modal fusion, this implementation introduces a high-low frequency feature adaptive enhancement module (HLFAE) in each decoding layer of the decoder module. For example... Figure 3 As shown, each decoding layer includes:

[0081] The high- and low-frequency feature separation section is used to separate the input features. Perform average pooling to obtain low-frequency components. Then, the low-frequency component is upsampled, and the input feature is subtracted from the upsampled low-frequency component to obtain the high-frequency component. It can be represented as: ;in, For average pooling operation, This is an upsampling operation.

[0082] The low-frequency enhancement section employs a lightweight self-attention approximation mechanism to enhance the nonlocal structural information of the low-frequency components, resulting in enhanced low-frequency features. Specifically, it first uses... Deep convolution extracts local structure, and then introduces the variance statistics of low-frequency features as global control information to improve the perception of the overall geometric layout of the scene. Let Total number of pixels Low-frequency components The Middle pixel value, If the mean is , then the variance is . Defined as: ; Variance through After convolutional mapping, the data is fused with structural features to achieve adaptive control of nonlocal representations, resulting in enhanced low-frequency features. .

[0083] The high-frequency enhancement section employs multi-scale dilated convolution branches to extract contextual information from different receptive fields of the high-frequency components in parallel, resulting in enhanced high-frequency features. This implementation sets up multiple groups... For dilated convolution, the dilation rate can be taken as... The enhanced high-frequency features are obtained by concatenating the outputs of each branch and then fusing them through convolution. .

[0084] The stitching and fusion section is used to integrate the enhanced low-frequency features. and enhanced high-frequency features The data is concatenated, weighted by convolutional layers and channel attention, and then connected to the input features via residual connections. The fused output is sent to the next decoding layer and can be represented as: ,in, This is the output for the splicing and fusion part.

[0085] The above separation-enhancement-fusion process can avoid mutual interference between high and low frequency information during decoding, and improve the recovery quality of both contour structure and edge details.

[0086] To prevent the encoder from losing input information during cross-domain training when training the cross-spectral depth estimation network, this implementation introduces a reconstruction network to reconstruct features back into the input image. Constraints, network reconstruction losses It can be represented as:

[0087] ;

[0088] in, This is the thermal infrared image of the left eye. This is the visible light image of the right eye. It has thermal infrared characteristics. It has visible light characteristics. For infrared reconstruction network, This is a visible light reconstruction network. This reconstruction network is only used for training constraints and does not participate in inference output.

[0089] The transspectral edge smoothing loss of this transspectral depth estimation network can be expressed as:

[0090] ;

[0091] Among them, among them, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by the transspectral depth estimation network, respectively.

[0092] The cross-spectral left-right consistency loss of this cross-spectral depth estimation network can be expressed as:

[0093] ;

[0094] in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map is obtained by reprojecting it onto the left view coordinate system.

[0095] The loss across spectral stages can be expressed as:

[0096] ;

[0097] in, The weights for the cross-spectral edge smoothing loss, The weights for cross-spectral consistency loss.

[0098] This implementation further constrains cross-modal geometric consistency through cross-spectral depth cyclic consistency, enabling the acquisition of stable and reliable geometric learning signal pairs and cross-spectral stereo pairs even without relying on cross-spectral photometric consistency. and its predicted parallax Construct new image pairs with swapped spectra through view transformation:

[0099] ;

[0100] Will New predictions are obtained by inputting the transspectral depth estimation network. And consistent with the original prediction, this cross-spectral depth cyclic consistency loss Represented as:

[0101] .

[0102] In summary, the loss function of the cross-spectral depth estimation network It can be represented as:

[0103] ;

[0104] in, Let be the weights for the cross-spectral depth cyclic consistency loss. Then the total loss of the two-stage network is expressed as:

[0105] .

[0106] This implementation still exhibits good parallax results in scenarios where visible light is unreliable. Figure 4The parallax visualization results for the exposed scene and the normal scene respectively show that the method of this embodiment still has good depth estimation ability in the unreliable visible light scene.

[0107] A second embodiment of the present invention relates to a transspectral stereo depth estimation device, comprising:

[0108] The acquisition module is used to acquire the left eye thermal infrared image and the right eye visible light image;

[0109] An estimation module is used to input the left-eye thermal infrared image and the right-eye visible light image into a transspectral depth estimation network to obtain a disparity map; wherein, the transspectral depth estimation network includes:

[0110] The encoder module is used to extract features from the left-eye thermal infrared image and the right-eye visible light image respectively to obtain thermal infrared features and visible light features;

[0111] The guided fusion module is used to generate a pixel-level reliability weight map based on the right eye visible light image, and to fuse the visible light features and thermal infrared features using the reliability weight map to obtain cross-modal fusion features;

[0112] The decoder module is used to separate the high-frequency and low-frequency features of the cross-modal fusion, enhance and stitch the obtained high-frequency and low-frequency features respectively, and obtain a disparity map based on the stitched enhanced features.

[0113] The encoder module includes:

[0114] A visible light encoder is used to extract features from the right eye's visible light image to obtain visible light features;

[0115] An infrared optical encoder is used to extract features from the left eye thermal infrared image to obtain thermal infrared features.

[0116] The parameters of the visible light encoder are obtained based on training of a self-supervised visible light basic network. The self-supervised visible light basic network adopts an encoder-decoder architecture, with visible light stereo pairs as inputs and left and right parallax as outputs.

[0117] The loss function of the self-supervised visible light basic network is expressed as:

[0118] ;

[0119] in, For the loss of self-supervised visible light basic networks, and Both are losses of photometric uniformity, expressed as: , Number of pixels and The positions in the left and right views are respectively pixel values, To obtain the position in the left view by reprojecting the right view using the left disparity predicted by a self-supervised visible light fundamental network, the position is... pixel values, To obtain the position in the right view by reprojecting the right disparity predicted by the left view using a self-supervised visible light fundamental network, the position is... pixel values, For similarity calculation, These are the weighting coefficients; and Both are edge-aware smoothing losses, expressed as: ,in, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by a self-supervised visible light-based network, respectively. and These are the left view and the right view, respectively. and Both are left-right consistency losses, expressed as: ,in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map obtained by reprojecting onto the left view coordinate system; The weights for edge-aware smoothing loss are... The weights for the left-right consistency loss.

[0120] The guided fusion module includes:

[0121] The depth estimation section is used to perform depth estimation on the right eye visible light image to obtain the prediction variance of the visible light pixels;

[0122] The reliability attention generation section is used to generate a pixel-level reliability weight map based on the prediction variance of the visible light pixels.

[0123] The fusion section is used to guide the visible light features using the pixel-level reliability weight map, and then fuse the guided visible light features with the thermal infrared features to obtain cross-modal fused features.

[0124] The decoder module includes several decoding layers, and each decoding layer includes:

[0125] The high- and low-frequency feature separation section is used to perform average pooling on the input features to obtain low-frequency components, then upsample the low-frequency components, and subtract the upsampled low-frequency components from the input features to obtain high-frequency components.

[0126] The low-frequency enhancement part is used to enhance the nonlocal structural information of the low-frequency components using a lightweight self-attention approximation mechanism to obtain enhanced low-frequency features;

[0127] The high-frequency enhancement part is used to extract contextual information of different receptive fields in parallel from the high-frequency components using multi-scale dilated convolution branches, so as to obtain enhanced high-frequency features.

[0128] The splicing and fusion section is used to splice the enhanced low-frequency features and enhanced high-frequency features, which are then weighted by convolutional layers and channel attention, and fused with the input features through residual connections before being output to the next level of decoding layer.

[0129] The loss function of the transspectral depth estimation network is expressed as:

[0130] ;

[0131] in, The loss function for the cross-spectral depth estimation network, The loss across spectral stages is expressed as: , The network reconstruction loss is represented as: ,in, This is the thermal infrared image of the left eye. This is the visible light image of the right eye. It has thermal infrared characteristics. It has visible light characteristics. For infrared reconstruction network, Reconstructing the network for visible light; and The loss for smoothing across spectral edges is expressed as: ,in, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by the transspectral depth estimation network, respectively. and The cross-spectral left-right consistency loss is expressed as: ,in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map obtained by reprojecting it onto the left view coordinate system. The weights for the cross-spectral edge smoothing loss, The weights for cross-spectral left-right consistency loss; The cross-spectral depth cyclic consistency loss is expressed as: , and The image pairs are respectively Input the left and right disparities obtained from the transspectral depth estimation network. This is a left-eye visible light image constructed from the right-eye visible light image and the left disparity predicted by a transspectral depth estimation network. The thermal infrared image of the right eye is constructed based on the left eye thermal infrared image and the right disparity predicted by the transspectral depth estimation network. The weights for cross-spectral depth cyclic consistency loss.

[0132] It is easy to see that this invention guides cross-spectral feature fusion with a pixel-level reliability weight map, which can adaptively identify and suppress noise interference in visible light degradation areas, fully preserve visible light texture details in well-lit areas, improve the robustness of cross-modal feature fusion, and avoid negative transfer problems. The decoder in this invention performs explicit high- and low-frequency separation and targeted enhancement on the fused features, effectively improving the global structure recovery capability dominated by thermal infrared and the edge texture detail restoration capability dominated by visible light, solving the problems of high- and low-frequency information mixing and detail loss in traditional decoding. The modular design of the cross-spectral depth estimation network of this invention is compatible with binocular cross-spectral imaging systems. The inference process does not rely on the ground truth depth value, and can make full use of the illumination robustness of thermal infrared and the texture richness of visible light, greatly improving the accuracy and stability of depth estimation in all-weather scenarios, and adapting to the practical application needs of autonomous driving, drone navigation, etc.

[0133] A third embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the transspectral stereo depth estimation method of the first embodiment.

[0134] The fourth embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the transspectral stereo depth estimation method of the first embodiment.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-spectral stereo depth estimation method, characterized in that, Includes the following steps: Acquire thermal infrared images from the left eye and visible light images from the right eye; The left-eye thermal infrared image and the right-eye visible light image are input into a transspectral depth estimation network to obtain a disparity map; wherein, the transspectral depth estimation network includes: The encoder module is used to extract features from the left-eye thermal infrared image and the right-eye visible light image respectively to obtain thermal infrared features and visible light features; The guided fusion module is used to generate a pixel-level reliability weight map based on the right eye visible light image, and to fuse the visible light features and thermal infrared features using the reliability weight map to obtain cross-modal fusion features; The decoder module is used to separate the high-frequency and low-frequency features of the cross-modal fusion, enhance and stitch the obtained high-frequency and low-frequency features respectively, and obtain a disparity map based on the stitched enhanced features.

2. The cross-spectral stereo depth estimation method of claim 1, wherein, The encoder module includes: A visible light encoder is used to extract features from the right eye's visible light image to obtain visible light features; An infrared optical encoder is used to extract features from the left eye thermal infrared image to obtain thermal infrared features. The parameters of the visible light encoder are obtained based on training of a self-supervised visible light basic network. The self-supervised visible light basic network adopts an encoder-decoder architecture, with visible light stereo pairs as inputs and left and right parallax as outputs.

3. The cross-spectral stereo depth estimation method of claim 2, wherein, The loss function of the self-supervised visible light basic network is expressed as: ； in, For the loss of self-supervised visible light basic networks, and Both are losses of photometric uniformity, expressed as: , Number of pixels and The positions in the left and right views are respectively pixel values, To obtain the position in the left view by reprojecting the right view using the left disparity predicted by a self-supervised visible light fundamental network, the position is... pixel values, To obtain the position in the right view by reprojecting the right disparity predicted by the left view using a self-supervised visible light fundamental network, the position is... pixel values, For similarity calculation, These are the weighting coefficients; and Both are edge-aware smoothing losses, expressed as: ,in, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by a self-supervised visible light-based network, respectively. and These are the left view and the right view, respectively. and Both are left-right consistency losses, expressed as: ,in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map obtained by reprojecting onto the left view coordinate system; The weights for edge-aware smoothing loss are... The weights for the left-right consistency loss.

4. The transspectral stereo depth estimation method according to claim 1, characterized in that, The guided fusion module includes: The depth estimation section is used to perform depth estimation on the right eye visible light image to obtain the prediction variance of the visible light pixels; The reliability attention generation section is used to generate a pixel-level reliability weight map based on the prediction variance of the visible light pixels. The fusion section is used to guide the visible light features using the pixel-level reliability weight map, and then fuse the guided visible light features with the thermal infrared features to obtain cross-modal fused features.

5. The transspectral stereo depth estimation method according to claim 1, characterized in that, The decoder module includes several decoding layers, and each decoding layer includes: The high- and low-frequency feature separation section is used to perform average pooling on the input features to obtain low-frequency components, then upsample the low-frequency components, and subtract the upsampled low-frequency components from the input features to obtain high-frequency components. The low-frequency enhancement part is used to enhance the nonlocal structural information of the low-frequency components using a lightweight self-attention approximation mechanism to obtain enhanced low-frequency features; The high-frequency enhancement part is used to extract contextual information of different receptive fields in parallel from the high-frequency components using multi-scale dilated convolution branches, so as to obtain enhanced high-frequency features. The splicing and fusion section is used to splice the enhanced low-frequency features and enhanced high-frequency features, which are then weighted by convolutional layers and channel attention, and fused with the input features through residual connections before being output to the next level of decoding layer.

6. The transspectral stereo depth estimation method according to claim 1, characterized in that, The loss function of the transspectral depth estimation network is expressed as: ； in, The loss function for the cross-spectral depth estimation network, The loss across spectral stages is expressed as: , The network reconstruction loss is represented as: ,in, This is the thermal infrared image of the left eye. This is the visible light image of the right eye. It has thermal infrared characteristics. It has visible light characteristics. For infrared reconstruction network, Reconstructing the network for visible light; and The loss for smoothing across spectral edges is expressed as: ,in, and These represent the gradients of the image in the x and y directions, respectively. and These are the left and right disparities predicted by the transspectral depth estimation network, respectively. and The cross-spectral left-right consistency loss is expressed as: ,in, This indicates the use of the left view to predict disparity. Predicting disparity in the right view The corresponding disparity map obtained by reprojecting it onto the left view coordinate system. The weights for the cross-spectral edge smoothing loss, The weights for cross-spectral left-right consistency loss; The cross-spectral depth cyclic consistency loss is expressed as: , and The image pairs are respectively Input the left and right disparities obtained from the transspectral depth estimation network. This is a left-eye visible light image constructed from the right-eye visible light image and the left disparity predicted by a transspectral depth estimation network. The thermal infrared image of the right eye is constructed based on the left eye thermal infrared image and the right disparity predicted by the transspectral depth estimation network. The weights for cross-spectral depth cyclic consistency loss.

7. A transspectral stereo depth estimation device, characterized in that, include: The acquisition module is used to acquire the left eye thermal infrared image and the right eye visible light image; An estimation module is used to input the left-eye thermal infrared image and the right-eye visible light image into a transspectral depth estimation network to obtain a disparity map; wherein, the transspectral depth estimation network includes: The encoder module is used to extract features from the left-eye thermal infrared image and the right-eye visible light image respectively to obtain thermal infrared features and visible light features; The guided fusion module is used to generate a pixel-level reliability weight map based on the right eye visible light image, and to fuse the visible light features and thermal infrared features using the reliability weight map to obtain cross-modal fusion features; The decoder module is used to separate the high-frequency and low-frequency features of the cross-modal fusion, enhance and stitch the obtained high-frequency and low-frequency features respectively, and obtain a disparity map based on the stitched enhanced features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the transspectral stereo depth estimation method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the transspectral stereo depth estimation method as described in any one of claims 1-6.

Images