Depth map super-resolution reconstruction method and system based on cross-modal feature guidance

Abstract

The invention provides a depth map super-resolution reconstruction method and system based on cross-modal feature guidance, and belongs to the technical field of deep learning, and the method comprises the steps: S1, signal collection and preprocessing; s2, shallow layer feature extraction and enhancement; s3, guiding and enhancing cross-modal features; s4, feature selection and adaptive weight distribution are carried out; s5, carrying out cross-modal attention fusion and reconstruction; and S6, optimizing the model. According to the method, the advantages of respective models of the Transform and the CNN are exerted, and respective shortages are supplemented, so that the super-resolution reconstruction quality of the depth map is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and system for super-resolution reconstruction of depth maps based on cross-modal feature guidance. Background Art

[0002] Depth map super-resolution reconstruction is an important and challenging task in image processing. Its goal is to generate higher-resolution and more detailed depth images from low-resolution depth images. In recent years, it has become a research hotspot in image processing and computer vision. Depth map super-resolution reconstruction methods are mainly categorized into filtering-based methods, optimization-based methods, and learning-based methods. Learning-based methods have garnered significant attention from both academia and industry due to their significant advantages over the other two methods in terms of computational speed and reconstruction quality.

[0003] Many existing depth map super-resolution reconstruction methods tend to adopt learning-based depth map super-resolution methods to significantly improve reconstruction performance, but convolutional neural networks mainly capture local information. To obtain global guided dependencies, it is usually necessary to stack multiple layers of convolution and pooling operations to gradually expand the receptive field, which cannot effectively establish cross-modal global associations. The interference of redundant textures in color images may cause artifacts in the reconstructed depth map. The visual Transformer benefits from the global modeling capability of the self-attention mechanism and can explicitly capture long-distance dependencies within the entire image, paying more attention to key areas, thereby improving the performance of related task models. However, its modeling of local information is relatively weak. The network divides the input image into image blocks of fixed size and introduces position information between image blocks through position encoding, but position encoding may ignore some local associations. Summary of the Invention

[0004] The present invention provides a depth map super-resolution reconstruction method and system based on cross-modal feature guidance, which leverages the advantages of the Transformer and CNN models and makes up for their respective shortcomings to improve the quality of depth map super-resolution reconstruction.

[0005] To achieve the above object, the present invention adopts the following technical solutions: The depth map super-resolution reconstruction method based on cross-modal feature guidance includes: S1. Signal acquisition and preprocessing; Acquire high-resolution color images of N images ; Collection and The corresponding low-resolution depth map and the true depth map ;Will After bicubic interpolation to the target resolution, the target resolution depth map is obtained ; S2. Shallow feature extraction and enhancement; right Perform 3×3 convolution and PReLU activation function processing to extract and The shallow features of the color information are extracted by using the residual group to enhance the features. Extract features from shallow structures and deep information ; S3. Cross-modal feature guidance and enhancement; Will The useful structural features in In the process, deep structural features are reconstructed; deep features are and color features The output of the cross-modal feature guidance module is fed into the cross-modal feature guidance module, and then the residual group is used to perform feature enhancement on the output of the cross-modal feature guidance module to obtain the guided reconstructed and enhanced color feature signal. and deep feature signals ,Will and Repeat the above process to obtain 、 ,in ; S4. Feature selection and adaptive weight allocation; the color feature signal and deep feature signals The input is sent to the feature selection module, and the channels containing high-frequency structural edges in the two features are given higher weights through cross-modal attention, thereby obtaining the feature signal ; S5. Cross-modal attention fusion and reconstruction; color feature signals and deep feature signals The characteristic signal obtained by upsampling and the characteristic signal obtained by adaptive selection The two modules are input into the cross-modal attention fusion module, and the attention mechanism is used to achieve the fusion and reconstruction of different modal information to obtain feature signals at all levels. Finally, the feature signals at all levels are passed through a 1×1 convolution layer and combined with the target resolution depth map. Add pixel by pixel to get the final reconstructed result signal, that is, the output depth map ; S6. Model optimization; to improve the output depth map and the true depth map The similarity between them is calculated by using the mean absolute error loss function to optimize the network. The loss function expression is: ; ; m is the number of pixel samples; L1 is the mean absolute error loss function regularization; Loss is the loss function.

[0006] In this specification, S3 includes: S31. Use strided convolution to transform the input and Downsampling to obtain and , and then use the feature alignment network to align the color features and depth features to obtain the aligned color features ; S32. Use DSConv to transform deep features Mapping to query space , align the color features Mapping to keyspace Sum value space , and then the obtained 、 and use The convolution kernel is expanded to obtain 、 and ,in, ; and is the spatial size after color feature downsampling; 、 and They are height, width, and number of channels; S33. Pass and Calculate the similarity between depth features and color features, from The edge structure information required for deep feature reconstruction is queried in the image, and the useful color structure information is transferred to the deep feature. ; S34. By and After superposition, a spatial attention mechanism is used for fusion projection, which can enhance the representation strength of color structure features by utilizing the initial reconstructed deep structure features and aligning the structural information in the color features, thereby reducing the impact of redundant texture in the color features on subsequent feature guidance and fusion. S35. The color feature is upsampled to the input space size and the residual connection is used to obtain the output , the deep features are adjusted by DSConv, upsampled to the input size and the residual is output ; S36. Deep features output by the cross-modal feature guidance module and color features Use RG to enhance features respectively, and obtain the color features and depth features after guided reconstruction and enhancement. and .

[0007] In this specification, S31 includes: S311. A spatial attention mechanism SA is used to capture the correlation between the structural distribution of the two features, and the offset information and mask weight are adjusted according to the structural information of the depth feature and the color feature to increase the accuracy of feature alignment. Convolution processes the output of SA to obtain the offset weight and mask weights ; S312. 、 And the features obtained after downsampling the color features input to the cross-modal feature guidance module In the input deformable convolution, Adjust the structural position distribution to obtain the aligned color features .

[0008] In this specification, S4 includes: S41.Yes Upsampled and then A simple projection using a 3×3 convolution is obtained and ; S42. and By superimposing the features in the channel dimension, the channels containing high-frequency structural edges in the two features are given higher weights through cross-modal attention, thereby improving the network's ability to reconstruct high-frequency information and obtaining the corresponding channel weights. ; S43. The accurate positioning of high-frequency structural information in the input features is included, and the weight information and the corresponding input features are dot-producted element by element to adaptively select useful features. The selected features are then superimposed and the channels are adjusted through convolution to obtain the output features. .

[0009] In this specification, S5 includes: S51. The three modal features of the input are projected by convolution with a kernel of 1 and activation function PReLu to obtain preliminary projection features. 、 and ; S52. By superposition 、 and Three features are then calculated using cross-modal attention to evaluate the correlation between cross-modal features. ; S53. When extracting spatial attention weights, in order to avoid the loss of high-frequency information, the inter-modal spatial attention module is used to calculate the independent SA weight of each feature. This module performs maximum and mean processing on each feature in the channel dimension, and then superimposes all the obtained spatial feature maps, performs convolution adjustment, and applies Sigmoid function to obtain the inter-modal spatial attention weights. ; S54. Use what you have learned and The weight is multiplied element by element with each mode to filter out useless information, and the two weighted selected features are added element by element to enhance the information, and the result is 、 and ; Then the three features after attention alignment calibration are superimposed in the channel dimension and output after convolution adjustment and RG enhancement ; S55. After a 1×1 convolution layer, and the bicubic interpolation Perform pixel-by-pixel addition to obtain the final reconstruction result .

[0010] A depth map super-resolution reconstruction system based on cross-modal feature guidance, applying any one of the above-mentioned depth map super-resolution reconstruction methods based on cross-modal feature guidance, the depth map super-resolution reconstruction system based on cross-modal feature guidance comprising: The cross-modal feature guidance module is used to transfer the spatial structure information of interest to deep reconstruction from the color features aligned by deformable convolution using the Transformer after the features obtained after residual group enhancement and refinement, and use the reconstructed depth features to enhance the color structure strength; A feature selection module is used to select channels that retain more spatial structure information from the color features and depth features obtained by the cross-modal feature guidance module; The cross-modal attention fusion module is used to fuse the depth and color semantic information at different levels of the network that is conducive to structural information reconstruction based on the attention mechanism, thereby improving the reconstruction effect of the depth map.

[0011] In this specification, the cross-modal feature guidance module includes two strided convolutions for downsampling, a feature alignment network for aligning the input in spatial structure, five DSConvs consisting of a 3×3 depth-wise convolution and a 1×1 convolution, three unfolding technologies Unfold for splitting the feature map and reshaping it into multiple vectors and a corresponding folding operation fold, a spatial attention mechanism SA and two transposed convolutions for upsampling.

[0012] In this specification, the feature alignment network includes a spatial attention mechanism SA, a 3×3 convolution layer and a deformable convolution.

[0013] In this specification, the feature selection module includes a transposed convolution for upsampling, two 3×3 convolutional layers, a cross-modal attention and a 1×1 convolutional layer.

[0014] In this specification, the cross-modal attention fusion module includes three 1×1 convolutional layers and an activation function PReLU, an inter-modal spatial attention and a cross-modal attention, a 1×1 convolutional layer for superimposing and enhancing the three features after attention alignment and calibration in the channel dimension, and a residual group.

[0015] In summary, the present invention has at least the following beneficial effects: This paper leverages the strengths of both Transformer and CNN models, addressing their respective shortcomings to improve the quality of super-resolution depth map reconstruction. Through a three-stage architecture design of cross-modal feature guidance, selection, and fusion, it enhances reconstruction quality and provides an effective solution for scenarios requiring high-precision depth maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 Schematic diagram of the process of depth map super-resolution reconstruction method based on cross-modal feature guidance involved in the present invention.

[0018] Figure 2 Schematic diagram of the topological structure of the depth map super-resolution reconstruction system based on cross-modal feature guidance involved in the present invention.

[0019] Figure 3 Schematic diagram of the topological structure of the cross-modal feature guidance module involved in the present invention.

[0020] Figure 4 Schematic diagram of the topological structure of the feature alignment network involved in the present invention.

[0021] Figure 5 Schematic diagram of the topological structure of the feature selection module involved in the present invention.

[0022] Figure 6 Schematic diagram of the topological structure of the cross-modal attention fusion module involved in the present invention. DETAILED DESCRIPTION

[0023] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0024] The disclosure below provides many different embodiments or examples for implementing different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the components and configurations of specific examples are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. In addition, the embodiments of the present invention may repeat reference numerals and / or reference letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] like Figure 1 The process shown in the figure, this embodiment provides a depth map super-resolution reconstruction method based on cross-modal feature guidance, including: S1. Signal acquisition and preprocessing; Acquire high-resolution color images of N images ; Collection and The corresponding low-resolution depth map and the true depth map ;Will After bicubic interpolation to the target resolution, the target resolution depth map is obtained ; S2. Shallow feature extraction and enhancement; right Perform 3×3 convolution and PReLU activation function processing to extract and The shallow features of the color information are extracted by using the residual group to enhance the features. Extract features from shallow structures and deep information ; S3. Cross-modal feature guidance and enhancement; Will The useful structural features in In the process, deep structural features are reconstructed; the residual group is used to enhance the features of the depth branch and the color branch respectively, and the color feature signal and the depth feature signal after guided reconstruction and enhancement are obtained; S4. Feature selection and adaptive weight allocation: Input the color feature signal and the depth feature signal into the multimodal feature fusion reconstruction module, and adaptively select the feature signal through the attention mechanism; S5. Cross-modal attention fusion and reconstruction: The feature signals obtained by upsampling the color feature signal and the depth feature signal and the feature signals obtained by adaptive selection are input into the cross-modal attention fusion module. The attention mechanism is used to achieve fusion and reconstruction between different modal information to obtain feature signals at all levels. Finally, the feature signals at all levels are passed through a 1×1 convolution layer and combined with the target resolution depth map. The final reconstructed result signal (SR depth map) is obtained by pixel-by-pixel addition, that is, the output depth map ; S6. Model optimization; to improve the output depth map and the true depth map The similarity between them is calculated by using the mean absolute error loss function to optimize the network. The loss function expression is: ; .

[0027] In some embodiments, in S1, a training set is established based on a depth map super-resolution reconstruction system guided by cross-modal features to obtain high-resolution color images of N images. and the low-resolution depth map corresponding to the high-resolution color image of the N images and the true depth map , where N is an integer greater than 1; the low-resolution depth map Depth map of target resolution is obtained by bicubic interpolation .

[0028] In some embodiments, the high-resolution color image is and low-resolution depth maps The shallow features of the HR color image and LR depth map are extracted through a 3×3 convolution and activation function PReLu, and the residual group is used for feature enhancement to obtain the shallow structure extraction features of the color information and depth information. and .

[0029] In some embodiments, S3 iteratively uses A cross-modal feature guidance module transfers useful structural features in color features to depth features. The cross-modal feature guidance module can make full use of the guidance features to reconstruct depth structural features. After the cross-modal feature guidance module is processed, the depth branch and color branch use RG for feature enhancement respectively, and the color features and depth features after guidance reconstruction and enhancement are obtained. and n is a hyperparameter (i.e., the number of cross-modal feature guidance modules used), which can be preferably set to 3.

[0030] In some embodiments, in order to select the more interesting part of the depth map reconstruction task from the two modal features in S4 and further retain the detailed information, a feature selection module based on the attention mechanism is used to give a greater weight to the more useful feature parts, thereby improving the information selection ability of the network. and Input into the multi-modal feature fusion reconstruction module, and adaptively select features through the attention mechanism .

[0031] In some embodiments, the color feature obtained by the feature guide portion is converted into , deep features Features obtained by upsampling and adaptive selection features The information is input into the cross-modal attention fusion module, and the attention mechanism is used to achieve the fusion and reconstruction of different modal information to obtain features at all levels. Finally, After a 1×1 convolution layer, and the bicubic interpolation Perform pixel-by-pixel addition to obtain the final reconstruction result .

[0032] In some embodiments, S3 includes: S31. Through strided convolution, the input and Downsample, and then use the feature alignment network to align the color features and depth features to obtain ; Here and , is to take what has been obtained and The feature is passed into the cross-modal feature guidance module and output is obtained, but the feature obtained in the previous step will be passed into the next cross-modal feature guidance module and this operation will be repeated. and Unified representation of the features obtained in the previous stage of input.

[0033] S32. Use DSConv to map deep features to query space ( ), mapping the color alignment features to the key space ( ) and value space ( ), and then the obtained 、 and use The convolution kernel is expanded to obtain 、 and ,in, ; and is the spatial size after color feature downsampling; 、 and They are height, width, and number of channels; S33. Pass and Calculate the similarity between depth features and color features, from The edge structure information required for deep feature reconstruction is queried in the image, and the useful color structure information is transferred to the deep feature. ; S34. In order to enhance the representation strength of color structure features and reduce the influence of redundant texture in color features on subsequent feature guidance and fusion, and After superposition, a spatial attention mechanism is used for fusion projection, and the structural information in the color features is enhanced and aligned using the initially reconstructed deep structural features; S35. The color feature is upsampled to the input space size and the residual connection is used to obtain the output , while the deep features are adjusted by DSConv, upsampled to the input size and residual to obtain the output ; S36. After the cross-modal feature guidance module is processed, the deep branch features and colored branch features Use RG to enhance features respectively, and obtain the color features and depth features after guided reconstruction and enhancement. and .

[0034] In some embodiments, S31 includes: S311. A spatial attention mechanism SA is used to capture the correlation between the structural distribution of the two features, and the offset information and mask weight are adjusted according to the structural information of the depth feature and the color feature to increase the accuracy of feature alignment. Convolution processes the output of SA to obtain the offset weight and mask weights ; S312. 、 The features obtained by downsampling the color features of the input cross-modal feature guidance module In the input deformable convolution, Adjust the structural position distribution to obtain the aligned color features .

[0035] In some embodiments, S4 includes: S41.Yes Upsampled and then A simple projection using a 3×3 convolution is obtained and ; S42. and By superimposing the features in the channel dimension, the channels containing high-frequency structural edges in the two features are given higher weights through cross-modal attention, thereby improving the network's ability to reconstruct high-frequency information and obtaining the corresponding channel weights. ; S43. The accurate positioning of high-frequency structural information in the input features is included, and the weight information and the corresponding input features are dot-producted element by element to adaptively select useful features. The selected features are then superimposed and the channels are adjusted through convolution to obtain the output features. .

[0036] In some embodiments, S5 includes: S51. The three modal features of the input are projected by convolution with a kernel of 1 and activation function PReLu to obtain preliminary projection features. 、 and ; S52. By superposition 、 and Three features are then calculated using cross-modal attention to evaluate the correlation between cross-modal features. ; S53. When extracting spatial attention weights, in order to avoid the loss of high-frequency information, the inter-modal spatial attention module is used to calculate the independent SA weight of each feature. This module performs maximum and mean processing on each feature in the channel dimension, and then superimposes all the obtained spatial feature maps, performs convolution adjustment, and applies Sigmoid function to obtain the inter-modal spatial attention weights. ; S54. Use what you have learned and The weight is multiplied element by element with each mode to filter out useless information, and the two weighted selected features are added element by element to enhance the information, and the result is 、 and ; Then the three features after attention alignment calibration are superimposed in the channel dimension and output after convolution adjustment and RG enhancement ; S55. After a 1×1 convolution layer, and the bicubic interpolation Perform pixel-by-pixel addition to obtain the final reconstruction result .

[0037] A depth map super-resolution reconstruction system based on cross-modal feature guidance applies any of the above-mentioned depth map super-resolution reconstruction methods based on cross-modal feature guidance. The depth map super-resolution reconstruction system based on cross-modal feature guidance includes a cross-modal feature guidance module, a feature selection module and a cross-modal attention fusion module.

[0038] like Figure 2 As shown, in the above implementation process, the residual group is used to further enhance and refine the feature map; the cross-modal feature guidance module is used to use Transformer to transfer the spatial structure information of interest to depth reconstruction from the color features after deformable convolution alignment using the features obtained after enhancement and refinement by the residual group, and use the reconstructed depth features to enhance the color structure strength; the feature selection module is used to select channels that retain more spatial structure information from the color features and depth features obtained by the cross-modal feature guidance module; the cross-modal attention fusion module is used to fuse the parts of the depth and color semantic information at different levels of the network that are conducive to structural information reconstruction based on the attention mechanism, thereby improving the reconstruction effect of the depth map.

[0039] Please see Figure 3 , Figure 3 A schematic diagram of the topological structure of the cross-modal feature guidance module provided in an embodiment of the present invention.

[0040] In some embodiments, the cross-modal feature guidance module includes two strided convolutions for downsampling, a feature alignment network for aligning the input in spatial structure, five DSConvs consisting of a 3×3 depthwise convolution and a 1×1 convolution, three unfolding techniques Unfold for splitting the feature map and reshaping it into multiple vectors and a corresponding folding operation fold, a spatial attention mechanism SA and two transposed convolutions for upsampling.

[0041] Please see Figure 4 , Figure 4 A schematic diagram of the topological structure of the feature alignment network provided in an embodiment of the present invention.

[0042] In some embodiments, the feature alignment network includes a spatial attention mechanism SA, a 3×3 convolution layer and a deformable convolution.

[0043] like Figure 3 and Figure 4 As shown, in the above implementation process, the input is transformed into and Down-sample, and then, in FAN and After channel dimension stacking, input SA. Then, use 3X3 convolution to process the output of SA to get the offset weight and mask weights Finally, 、 and In the input deformable convolution, Adjust the structural position distribution to obtain the aligned color features ; DSConv is then used to map the deep features to the query space ( ), mapping the color alignment features to the key space ( ) and value space ( ), and then the obtained 、 and use The convolution kernel is expanded to obtain 、 and ,in, ; and is the spatial size after color feature downsampling, 、 and They are The height, width and number of channels of and Calculate the similarity between depth features and color features, from The edge structure information required for deep feature reconstruction is queried in the image, and the useful color structure information is transferred to the deep feature. In order to enhance the representation strength of color structure features and reduce the influence of redundant texture in color features on subsequent feature guidance and fusion, and After superposition, the spatial attention mechanism is used for fusion projection, and the structural information in the color features is enhanced and aligned using the initially reconstructed deep structural features; the color features are output after upsampling to the input space size and residual connection. , while the deep features are adjusted by DSConv, upsampled to the input size and residual to obtain the output The process can be expressed as: ; ; in, For SA network; represents a deformable convolution operation; ; ; ; in, 、 and is the DSConv weight of the corresponding feature; Represents a convolution operation.

[0044] ; ; in, is the number of long positions; Represents the Softmax function; is the learnable scaling factor; is the weight of DSConv convolution; Represents a fold operation.

[0045] ; .

[0046] Please see Figure 5 , Figure 5 A schematic diagram of the topological structure of the feature selection module provided in an embodiment of the present invention.

[0047] In some embodiments, the feature selection module includes a transposed convolution for upsampling, two 3×3 convolutional layers, a cross-modal attention and a 1×1 convolutional layer.

[0048] like Figure 5 As shown, Upsampled and then A simple projection using a 3×3 convolution is obtained and ; then and By superimposing the features in the channel dimension, the channels containing high-frequency structural edges in the two features are given higher weights through cross-modal attention, thereby improving the network's ability to reconstruct high-frequency information and obtaining the corresponding channel weights. , The accurate positioning of high-frequency structural information in the input features is included, and the weight information and the corresponding input features are dot-producted element by element to adaptively select useful features. The selected features are then superimposed and the channels are adjusted through convolution to obtain the output features. .

[0049] ; ; ; is the marked channel weight; is the global mean pooling function; is a fully connected function; is the Sigmoid function.

[0050] .

[0051] Please see Figure 6 , Figure 6 Schematic diagram of the topological structure of the cross-modal attention fusion module provided in an embodiment of the present invention.

[0052] In some embodiments, the cross-modal attention fusion module includes three 1×1 convolutional layers and an activation function PReLU, an inter-modal spatial attention and a cross-modal attention, a 1×1 convolutional layer for superimposing and enhancing the three features after attention alignment and calibration in the channel dimension, and a residual group.

[0053] The three modal features of the input are projected by convolution with a kernel of 1 and activation function PReLu to obtain preliminary projection features. 、 and ; by superposition 、 and Three features are then calculated using cross-modal attention When extracting spatial attention weights, in order to avoid the loss of high-frequency information, the inter-modal spatial attention module is used to calculate the independent SA weight of each feature. This module performs maximum and mean processing on each feature in the channel dimension, and then superimposes all the obtained spatial feature maps, performs convolution adjustment, and performs Sigmoid function to obtain the inter-modal spatial attention weights. ; Use what you have learned and The weight is multiplied element by element with each mode to filter out useless information, and the two weighted selected features are added element by element to enhance the information, and the result is 、 and ; Then the three features after attention alignment calibration are superimposed in the channel dimension and output after convolution adjustment and RG enhancement ; Finally, After a 1×1 convolution layer, and the bicubic interpolation Perform pixel-by-pixel addition to obtain the final reconstruction result .

[0054] ; ; ; ; ; ; in For CMA module; Indicates the calculated CA weight; is the ISA weight; and It is the maximum and mean processing of channel dimension.

[0055] ; ; ; ; .

[0056] In some embodiments, the present invention uses real datasets and synthetic datasets to verify the effectiveness of the proposed model. In the real dataset, the first 1,000 RGB-D image pairs from the NYUv2 data are used as the training dataset. The training images are cropped into 256×256 image blocks, and the remaining 449 RGB-D image pairs are used as the test set. To verify the generalization of the real dataset training, the LuRGB-D dataset (6 pairs) and the MiddleburyRGB-D dataset (30 pairs) provided by Zhao et al. are used for generalization verification. In the synthetic dataset, 82 RGB-D image pairs are selected from the Middlebury dataset (34 pairs) and the MPISintel dataset (58 pairs) as the training set. The test set uses 6 RGB-D image pairs from the Middlebury dataset (2005). To increase the data volume and reduce computation, the RGB-D image pairs are cropped into 256×256 image blocks with a cropping stride of 64. This process is applied to all sampling multiples. The real depth edge image is extracted by the Prewitt operator, and the LR depth map is obtained by downsampling the real depth map through bicubic interpolation. During training, all image pairs are augmented using 90° rotation, 180° rotation, horizontal flip, and vertical flip.

[0057] After the training set is built, the model can be trained and tested on the pytorch framework. Set to 3; downsampling multiple g Set to 16; Set to 3; the number of self-attention heads h Set to 8; the initial learning rate is The number of feature channels is 64. The Adam optimization algorithm is used to optimize the network parameters, where the parameter settings of the Adam optimizer are 0.900, is 0.999, and the learning rate is . Root mean square error (RMSE) was used to evaluate model performance.

[0058] The present invention uses the NYU dataset, the Middlebury dataset, and the Lu dataset to verify the effectiveness of the proposed model. In the comparative experiment, the learning-based comparison methods include: DJFR, CUNet, FDKN, DKN, FDSR, DCTNet, JIIF, AHMF, DAGF, GeoDSR, and DSR-Diff. The experimental results are shown in Table 1, where TCFGN is the method proposed by the present invention. As can be seen from Table 1 (the optimal value is underlined), in most cases, the RSME of the present invention is the lowest, and the reconstruction effect is significantly better than other reconstruction methods. Table 1 Comparison of RMSE values ​​on three test sets.

[0059] Table 1. RMSE statistics for the NYU, Middlebury, and Lu datasets In summary, the embodiments of the present invention provide prior knowledge for the cross-modal attention fusion module through the cross-modal feature guidance module and the feature selection module, thereby improving the network's ability to reconstruct high-resolution depth maps with clearer edges and fewer artifacts.

[0060] The above embodiments are intended to illustrate the present invention, not to limit the present invention. Therefore, changes in illustrative values ​​or substitutions of equivalent components should still fall within the scope of the present invention.

[0061] From the above detailed description, it will be clear to those skilled in the art that the present invention can indeed achieve the aforementioned objectives and is in compliance with the provisions of the Patent Law.

[0062] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as covering the preferred embodiments and all changes and modifications that fall within the scope of the invention. The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

[0063] It should be noted that the above description of the relevant processes is for illustration and purpose only and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the processes under the guidance of this specification. However, such modifications and changes are still within the scope of this specification.

[0064] The basic concepts have been described above. It will be apparent to those skilled in the art after reading this application that the above disclosures are merely illustrative and do not constitute limitations on this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and amendments to this application. Such modifications, improvements, and amendments are suggested in this application and remain within the spirit and scope of the exemplary embodiments of this application.

[0065] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "one embodiment," "an embodiment," or "an alternative embodiment" mentioned twice or more in different places in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application may be appropriately combined.

[0066] Furthermore, those skilled in the art will appreciate that various aspects of the present application may be illustrated and described in terms of a number of patentable categories or situations, including any new and useful process, machine, product, or combination of substances, or any new and useful improvement thereof. Thus, various aspects of the present application may be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software. Each of the above hardware and software may be referred to as a "unit," "module," or "system." Furthermore, various aspects of the present application may take the form of a computer program product embodied in one or more computer-readable media, with computer-readable program code embodied therein.

[0067] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C programming language, Visual Basic, Fortran2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run entirely on the user's computer, or as a standalone software package on the user's computer, or partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0068] In addition, unless expressly stated in the claims, the order of the processing elements and sequences described in this application, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some embodiments of the invention that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a pure software solution, for example, installation on an existing server or mobile device.

[0069] Similarly, it should be noted that in order to simplify the presentation of this disclosure and thereby facilitate understanding of one or more of the invention's embodiments, the foregoing descriptions of the embodiments of this disclosure sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this approach should not be interpreted as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject matter of the invention may possess fewer features than the single embodiment described above.

Claims

1. A depth map super-resolution reconstruction method based on cross-modal feature guidance, characterized by: include: S1. Signal acquisition and preprocessing; Acquire high-resolution color images of N images ; Collection and The corresponding low-resolution depth map and the true depth map ;Will After bicubic interpolation to the target resolution, the target resolution depth map is obtained ; S2. Shallow feature extraction and enhancement; right Perform 3×3 convolution and PReLU activation function processing to extract and The shallow features of the color information are extracted by using the residual group to enhance the features. Extract features from shallow structures and deep information ; S3. Cross-modal feature guidance and enhancement; Will The structural features in the In , reconstruct the deep structural features; Deep Features and color features The output of the cross-modal feature guidance module is fed into the cross-modal feature guidance module, and then the residual group is used to perform feature enhancement on the output of the cross-modal feature guidance module to obtain the guided reconstructed and enhanced color feature signal. and deep feature signals ,Will and Repeat the above process to obtain 、 ,in , n is a hyperparameter; S4. Feature selection and adaptive weight allocation; the color feature signal and deep feature signals The input is sent to the feature selection module, and the channels containing high-frequency structural edges in the two features are given higher weights through cross-modal attention, thereby obtaining the feature signal ; S5. Cross-modal attention fusion and reconstruction; color feature signals and deep feature signals The characteristic signal obtained by upsampling and the characteristic signal obtained by adaptive selection The two modules are input into the cross-modal attention fusion module, and the attention mechanism is used to achieve the fusion and reconstruction of different modal information to obtain feature signals at all levels. Finally, the feature signals at all levels are passed through a 1×1 convolution layer and combined with the target resolution depth map. Add pixel by pixel to get the final reconstructed result signal, that is, the output depth map ; S6. Model optimization; to improve the output depth map and the true depth map The similarity between them is calculated by using the mean absolute error loss function to optimize the network. The loss function expression is: ; ; m is the number of pixel samples; L1 is the mean absolute error loss function regularization; Loss is the loss function.

2. The depth map super-resolution reconstruction method based on cross-modal feature guidance according to claim 1, characterized in that S3 include: S31. Use strided convolution to transform the input and Downsampling to obtain and , and then use the feature alignment network to align the color features and depth features to obtain the aligned color features ; S32. Use DSConv to transform deep features Mapping to query space , align the color features Mapping to keyspace Sum value space , and then the obtained 、 and use The convolution kernel is expanded to obtain 、 and ,in, ; and is the spatial size after color feature downsampling; 、 and They are height, width, and number of channels; S33. Pass and Calculate the similarity between depth features and color features, from The edge structure information required for deep feature reconstruction is queried in the image, and the useful color structure information is transferred to the deep feature. ; S34. By and After superposition, a spatial attention mechanism is used for fusion projection, which can enhance the representation strength of color structure features by utilizing the initial reconstructed deep structure features and aligning the structural information in the color features, thereby reducing the impact of redundant texture in the color features on subsequent feature guidance and fusion. S35. The color feature is upsampled to the input space size and the residual connection is used to obtain the output , the deep features are adjusted by DSConv, upsampled to the input size and the residual is output ; S36. Deep features output by the cross-modal feature guidance module and color features Use RG to enhance features respectively, and obtain the color features and depth features after guided reconstruction and enhancement. and .

3. The depth map super-resolution reconstruction method based on cross-modal feature guidance according to claim 2 is characterized in that S31 includes: S311. A spatial attention mechanism SA is used to capture the correlation between the structural distribution of the two features. The offset information and mask weight are adjusted according to the structural information of the depth feature and the color feature to increase the accuracy of feature alignment. The output of SA is processed using 3X3 convolution to obtain the offset weight. and mask weights ; S312. 、 And the features obtained after downsampling the color features input to the cross-modal feature guidance module In the input deformable convolution, Adjust the structural position distribution to obtain the aligned color features .

4. The depth map super-resolution reconstruction method based on cross-modal feature guidance according to claim 1, characterized in that S4 include: S41.Yes Upsampled and then A simple projection using a 3×3 convolution is obtained and ; S42. and By superimposing the features in the channel dimension, the channels containing high-frequency structural edges in the two features are given higher weights through cross-modal attention, thereby improving the network's ability to reconstruct high-frequency information and obtaining the corresponding channel weights. ; S43. The accurate positioning of high-frequency structural information in the input features is included, and the weight information and the corresponding input features are dot-producted element by element to adaptively select useful features. The selected features are then superimposed and the channels are adjusted through convolution to obtain the output features. .

5. The depth map super-resolution reconstruction method based on cross-modal feature guidance according to claim 1, characterized in that S5 include: S51. The three modal features of the input are projected by convolution with a kernel of 1 and activation function PReLu to obtain preliminary projection features. 、 and ; S52. By superposition 、 and Three features are then calculated using cross-modal attention to evaluate the correlation between cross-modal features. ; S53. When extracting spatial attention weights, in order to avoid the loss of high-frequency information, the inter-modal spatial attention module is used to calculate the independent SA weight of each feature. This module performs maximum and mean processing on each feature in the channel dimension, and then superimposes all the obtained spatial feature maps, performs convolution adjustment, and applies Sigmoid function to obtain the inter-modal spatial attention weights. ; S54. Use what you have learned and The weight is multiplied element by element with each mode to filter out useless information, and the two weighted selected features are added element by element to enhance the information, and the result is 、 and ; Then the three features after attention alignment calibration are superimposed in the channel dimension and output after convolution adjustment and RG enhancement ; S55. After a 1×1 convolution layer, and the bicubic interpolation Perform pixel-by-pixel addition to obtain the final reconstruction result .

6. A depth map super-resolution reconstruction system based on cross-modal feature guidance, characterized by: The method for super-resolution reconstruction of a depth map based on cross-modal feature guidance according to any one of claims 1 to 5 is applied, and the system for super-resolution reconstruction of a depth map based on cross-modal feature guidance comprises: The cross-modal feature guidance module is used to transfer the spatial structure information of interest to deep reconstruction from the color features aligned by deformable convolution using the Transformer after the features obtained after residual group enhancement and refinement, and use the reconstructed depth features to enhance the color structure strength; A feature selection module is used to select channels that retain more spatial structure information from the color features and depth features obtained by the cross-modal feature guidance module; The cross-modal attention fusion module is used to fuse the depth and color semantic information at different levels of the network that is conducive to structural information reconstruction based on the attention mechanism, thereby improving the reconstruction effect of the depth map.

7. The depth map super-resolution reconstruction system based on cross-modal feature guidance according to claim 6, characterized in that The cross-modal feature guidance module includes two strided convolutions for downsampling, a feature alignment network for aligning the input in spatial structure, five DSConvs consisting of a 3×3 depth-wise convolution and a 1×1 convolution, three unfolding techniques Unfold for splitting the feature map into multiple vectors and a corresponding folding operation fold, a spatial attention mechanism SA and two transposed convolutions for upsampling.

8. The depth map super-resolution reconstruction system based on cross-modal feature guidance according to claim 7, characterized in that: The feature alignment network consists of a spatial attention mechanism SA, a 3×3 convolutional layer and a deformable convolution.

9. The depth map super-resolution reconstruction system based on cross-modal feature guidance according to claim 6, characterized in that The feature selection module includes a transposed convolution for upsampling, two 3×3 convolutional layers, a cross-modal attention and a 1×1 convolutional layer.

10. The depth map super-resolution reconstruction system based on cross-modal feature guidance according to claim 6, characterized in that: The cross-modal attention fusion module includes three 1×1 convolutional layers and the activation function PReLU, an inter-modal spatial attention and a cross-modal attention, a 1×1 convolutional layer for superimposing and enhancing the three features after attention alignment and calibration in the channel dimension, and a residual group.

Images