Depth map super-resolution reconstruction method and system based on scale-aware implicit function

Abstract

The present invention proposes a method and system for super-resolution reconstruction of depth maps based on scale-aware implicit functions. The method extracts features from high-resolution guide maps and low-resolution depth maps to obtain high-resolution guide features and low-resolution depth features. The extracted features are then mutually fused using implicit cross-attention and updated in both directions. After multiple iterations, the final high-resolution guide features and low-resolution depth features are obtained. The high-resolution guide features and low-resolution depth features are decoded by a Transformer based on implicit cross-attention and implicit self-attention, and finally a depth map reconstructed with super-resolution of any scale is obtained. The present invention provides a unified solution for super-resolution reconstruction of depth maps of any scale. At the same time, the proposed model and system achieve the best results in subjective and objective evaluations compared with the most advanced methods on mainstream data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method and system for super-resolution reconstruction of a depth map based on a scale-aware implicit function. Background Art

[0002] RGB-D data including texture images and depth images are widely used in many tasks of 3D computer vision, such as virtual reality, 3D reconstruction, object tracking, etc. Nowadays, people can capture high-resolution (HR) color images with their mobile phones anytime and anywhere. However, there is still a lack of effective methods to obtain high-quality depth maps to record real scenes. Previous studies have inferred depth maps by stereo matching between multi-view color images, and textureless areas and occlusions are the biggest challenges. With the development of deep neural networks, monocular depth estimation from a single color image to obtain a depth map has become a new research trend. However, due to highly ill-posed formulas, this model still faces considerable difficulties in promotion. In this application scheme, alternative methods for capturing depth maps using consumer-grade sensors are considered, such as structured light cameras and time-of-flight cameras (ToF). Specifically, structured light sensors estimate depth maps by reflecting infrared dot patterns in the scene, while ToF sensors measure the depth value of each pixel of the image based on the phase difference between the emitted infrared light and the infrared light. Although sensor-based solutions are more flexible in depth estimation, the quality of the depth map cannot meet the requirements of downstream applications due to problems such as low resolution (LR) and significant noise.

[0003] In the early stage, inspired by single image super-resolution (SISR), many variants for single depth map super-resolution (SDSR) were proposed. However, since the required upsampling scale is always much larger than SISR, fine-grained details and tiny structures are either distorted or disappear in the reconstructed depth map. To alleviate the challenge of SDSR, cross-source information from describing the same scene to the depth map is introduced, such as high-resolution color images. Obviously, high-resolution color images contain more accurate object edges than low-resolution depth maps. The corresponding problem definition is the so-called guided depth map super-resolution (GDSR), and the research hotspot of GDSR has successively gone through the optimization and learning stages. Optimization-based methods use manual guidance to prioritize, for example, edge co-occurrence. Since manual models usually have a gap in the true joint distribution of depth maps and color images, learning-based manual models show more powerful model performance through data-driven thinking.

[0004] Recently, LIIF proposed the concept of implicit image function and learned a general SISR model for arbitrary upsampling scales. Interestingly, this flexible model even outperforms the model trained with a fixed scale and shows good results under out-of-distribution superscaling. This topic has attracted the attention of the low-level vision community. In contrast, there are fewer studies on GDSR based on implicit deep function learning because related studies face three problems.

[0005] First, the traditional scale-dependent model shows that the bidirectional fusion between deep features and texture features in the encoder has a significant performance gain. Therefore, it should be necessary to try to fuse deep features and color features in the model encoder based on implicit depth function. However, to realize the above attempt, how the model can effectively fuse the continuous upsampling scale information is a problem that needs to be solved. Obviously, upsampling layers, such as pooling and transposed convolution, are only designed for fixed upsampling scales.

[0006] Second, work based on implicit depth functions needs to verify the effect by introducing scale information into the depth function. However, the role of scale information in the encoder is still not fully utilized. Therefore, exploring the role and importance of scale information in the decoder is a problem that needs to be solved.

[0007] Third, in the decoder, the input has fused features from two sources. On the one hand, the guided features have the advantage of being in the high-resolution space, but contain a distribution shift to the depth domain. On the other hand, the depth features are more consistent with the true depth values, but have the disadvantage of low-resolution feature representation. Therefore, the third question is how to exploit these cross-domains in the decoder to effectively model position distance and content similarity. Summary of the invention

[0008] In view of the above situation, the main purpose of the present invention is to propose a depth map super-resolution reconstruction method and system based on scale-aware implicit function to solve the above technical problems.

[0009] The present invention proposes a depth map super-resolution reconstruction method based on scale-aware implicit function, the method comprising the following steps:

[0010] Step 1: Use convolution operation to obtain shallow guidance features of the high-resolution guidance map and shallow depth features of the low-resolution depth map;

[0011] Step 2: Based on Transformer, a sliding window is used to extract deep features from shallow guide features and shallow depth features respectively. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guide features and low-resolution depth features.

[0012] Step 3: The high-resolution guidance features and the low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain updated high-resolution guidance features and low-resolution depth features;

[0013] Step 4: Using the updated high-resolution guide features and low-resolution depth features as input, repeat steps 1 to 3 in an iterative manner to obtain the final high-resolution guide features and low-resolution depth features;

[0014] Step 5: Using an external implicit Transformer based on implicit cross attention, taking the final high-resolution guided features and low-resolution deep features as input, using the final high-resolution guided features to guide the final low-resolution deep features for decoding, and generating the first high-resolution deep features;

[0015] An internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution deep features as input to generate the second high-resolution deep features;

[0016] Step 6: After splicing the first high-resolution depth feature and the second high-resolution depth feature, send them to the fully connected layer to generate a high-resolution depth residual image; upsample the low-resolution depth image by bicubic interpolation to obtain the low-frequency component of the target depth map, add the high-resolution depth residual image and the low-frequency component of the target depth map to obtain a super-resolution reconstructed depth map of any scale.

[0017] The present invention also proposes a depth map super-resolution reconstruction system based on scale-aware implicit function, wherein the system applies the depth map super-resolution reconstruction method based on scale-aware implicit function as described above, and the system comprises:

[0018] Guided feature extraction branch for:

[0019] Use convolution operation to obtain shallow guidance features of high-resolution guidance map;

[0020] Based on Transformer, a sliding window method is used to extract deep features from shallow guiding features. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guiding features.

[0021] The deep super-resolution branch is used to:

[0022] Use convolution operation to obtain shallow depth features of low-resolution depth map;

[0023] Based on Transformer, a sliding window method is used to extract deep features from shallow and deep features respectively. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain low-resolution deep features.

[0024] Bidirectional implicit fusion block for:

[0025] The high-resolution guidance features and low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain updated high-resolution guidance features and low-resolution depth features;

[0026] Encoders for:

[0027] Taking the updated high-resolution guided features and low-resolution deep features as input, the guided feature extraction branch, the deep super-resolution branch and the bidirectional implicit fusion block are iteratively repeated to obtain the final high-resolution guided features and low-resolution deep features.

[0028] Decoder for:

[0029] An external implicit Transformer based on implicit cross attention is adopted, which takes the final high-resolution guided features and low-resolution deep features as input, and uses the final high-resolution guided features to guide the final low-resolution deep features for decoding to generate the first high-resolution deep features;

[0030] An internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution deep features as input to generate the second high-resolution deep features;

[0031] After splicing the first high-resolution depth feature and the second high-resolution depth feature, they are sent to the fully connected layer to generate a high-resolution depth residual image; the low-frequency component of the target depth map obtained by bicubic interpolation upsampling the low-resolution depth image is added to the high-resolution depth residual image and the low-frequency component of the target depth map to obtain a super-resolution reconstructed depth map of any scale.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The encoder of the present invention realizes the alignment of texture guidance features and depth features based on continuous coordinate mapping, and introduces implicit cross-attention to fuse the above-mentioned depth features and guidance features with continuous scale differences. This fusion process not only screens out irrelevant guidance features, but also refines the depth features, realizing the bidirectional fusion of guidance features and depth features.

[0034] 2. To improve the model’s perception of continuous reconstruction scale tasks, the encoder embeds scale information into the feedforward network and position encoding to generate a more refined and scale-sensitive feature representation, while the decoder realizes continuous scale depth map reconstruction based on the implicit attention mechanism of embedded scale information.

[0035] 3. The bidirectional implicit feature fusion module of the present invention is composed of a texture-guided depth refinement block and a depth-guided texture selection block, and performs implicit feature fusion and iterative update of different scales of guided features and depth features based on implicit cross-attention, which solves the problem that the encoder in the traditional deep super-resolution model based on implicit depth function ignores the fusion of high-resolution guided features and low-resolution depth features.

[0036] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a depth map super-resolution reconstruction method based on scale-aware implicit function proposed by the present invention;

[0038] Figure 2 This is the overall framework diagram of the depth map super-resolution reconstruction system based on scale-aware implicit function proposed by the present invention;

[0039] Figure 3 It is a structural schematic diagram of the bidirectional implicit fusion block of the present invention. DETAILED DESCRIPTION

[0040] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0041] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0042] See also Figure 1 This embodiment provides a depth map super-resolution reconstruction method based on scale-aware implicit function, the method comprising the following steps:

[0043] Step 1: Use convolution operation to obtain shallow guidance features of the high-resolution guidance map and shallow depth features of the low-resolution depth map;

[0044] Step 2: Based on Transformer, a sliding window is used to extract deep features from shallow guide features and shallow depth features respectively. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guide features and low-resolution depth features.

[0045] Furthermore, in step 2, based on Transformer, a sliding window is used to extract deep features from shallow guide features and shallow depth features respectively; and in the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guide features and low-resolution depth features. The process specifically includes the following steps:

[0046] Step 2.1, set the window size, and divide the input image into several image blocks according to the window size;

[0047] Step 2.2, map the image blocks into tokens using linear transformation, and obtain the relative coordinates and scale information between any token pairs;

[0048] Step 2.3, concatenate the relative coordinates and scale information, learn the scale information features through a sub-network composed of two MLP layers, and obtain the scale-aware position encoding;

[0049] Step 2.4: Introduce scale-aware position encoding to perform self-attention calculation on the tokens in each window to obtain scale-aware self-attention output;

[0050] Step 2.5: Repeat steps 2.1 to 2.4 using shallow guidance features and shallow depth features as input features to obtain high-resolution guidance features and low-resolution depth features.

[0051] Furthermore, the process of introducing scale-aware position encoding to calculate the self-attention of the tokens in each window has the following relationship:

[0052] ;

[0053] in, Represent query vector, key vector, and value vector respectively; represents the transpose operation, express Q and K The number of dimensions, represents a static positional encoding of the scale-free input, represents the dynamic position encoding adjusted by the dynamic scale information, represents the self-attention calculation operation, represents the normalized exponential function.

[0054] Step 3: The high-resolution guidance features and the low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain updated high-resolution guidance features and low-resolution depth features;

[0055] Furthermore, in step 3, the high-resolution guidance features and the low-resolution depth features are mutually fused through implicit cross attention and bidirectionally updated to obtain the updated high-resolution guidance features and low-resolution depth features, and the process specifically includes the following steps:

[0056] The low-resolution deep features are passed through a The convolutional layer gets the first query;

[0057] The high-resolution guidance features are passed through two independent The convolutional layer obtains the first key and the first value;

[0058] Establishing a mapping from a query position in low-resolution coordinates to a window center in high-resolution coordinates to determine a local window of a first key and a first value;

[0059] When mapping, select the adjacent integer position as the center of the local window, and upsample the first query according to the mapping relationship;

[0060] Performing position encoding using the relative coordinates between the first query and each element in the first key and the first value as target scale information to obtain a first scale-aware position encoding, performing implicit cross-attention calculation using the first query, the first key, and the first value, and introducing the first scale-aware position encoding in the attention calculation process to obtain a first implicit cross-attention output;

[0061] The first implicit cross-attention output is concatenated with the target scale information in the channel dimension to explicitly introduce the scale information;

[0062] The concatenated result is then fed into a feed-forward neural network consisting of two MLP layers to obtain updated low-resolution deep features.

[0063] Upsampling the updated low-resolution depth features through bilinear interpolation to achieve the same size as the high-resolution guidance features to obtain the upsampled depth features;

[0064] The upsampled depth features are multiplied by the corresponding coordinate elements of the high-resolution guidance features and passed through an MLP layer to obtain the updated high-resolution guidance features.

[0065] Furthermore, the first query, the first key, and the first value are used to perform implicit cross-attention calculation, and the first scale-aware position encoding is introduced in the attention calculation process to obtain the first implicit cross-attention output. The corresponding process has the following relationship:

[0066] ;

[0067] in, represents the implicit cross-attention computation operation, represents an element in the first query, Respectively represent the first query, the first key, and the first value; Relative coordinates. Represents a subnetwork containing three MLP layers.

[0068] Step 4: Using the updated high-resolution guide features and low-resolution depth features as input, repeat steps 1 to 3 in an iterative manner to obtain the final high-resolution guide features and low-resolution depth features;

[0069] Step 5: Using an external implicit Transformer based on implicit cross attention, taking the final high-resolution guided features and low-resolution deep features as input, using the final high-resolution guided features to guide the final low-resolution deep features for decoding, and generating the first high-resolution deep features;

[0070] An internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution deep features as input to generate the second high-resolution deep features;

[0071] Furthermore, in step 5, an external implicit Transformer based on implicit cross attention is used, the final high-resolution guided feature and the low-resolution depth feature are taken as input, and the final high-resolution guided feature is used to guide the final low-resolution depth feature for decoding, and generating the first high-resolution depth feature specifically includes the following steps:

[0072] The final low-resolution depth feature is passed through a The convolutional layer obtains the second query;

[0073] The final high-resolution guided features are passed through two independent The convolutional layer obtains the second key and the second value;

[0074] establishing a mapping from a query position in low-resolution coordinates to a window center in high-resolution coordinates to determine a local window for a second key and a second value;

[0075] When mapping, select the adjacent integer position as the center of the local window, and upsample the second query according to the mapping relationship;

[0076] Performing position encoding using the relative coordinates between the second query and each element in the second key and the second value as target scale information to obtain a second scale-aware position encoding, performing implicit cross-attention calculation using the second query, the second key, and the second value, and introducing the second scale-aware position encoding in the attention calculation process to obtain a second implicit cross-attention output;

[0077] The second implicit cross-attention output is concatenated with the target scale information in the channel dimension and decoded through two MLP layers to obtain the first high-resolution deep feature.

[0078] Furthermore, in step 5, an internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution depth feature as input, and generating the second high-resolution depth feature specifically includes the following steps:

[0079] The final low-resolution deep features are processed using three separate convolutional layers to generate the third query, the third key, and the third value;

[0080] Upsampling the third query to the high-resolution space by bilinear interpolation to perform high-resolution coordinate traversal;

[0081] The relative coordinates between the third query and each element in the third key and the third value after the above sampling, and the sine embedding is added to the relative coordinates to explicitly map the low-dimensional relative position to the high-dimensional representation. The process corresponding to the sine embedding added to the relative coordinates has the following relationship:

[0082] ;

[0083] in, represents the embedded output, represents a hyperparameter;

[0084] The upsampled third query is used to perform self-attention calculation with the third key and the third value, and the relative coordinates after additional sine embedding are used as position encoding. The position encoding is introduced in the attention calculation process to obtain implicit self-attention output.

[0085] The implicit self-attention output is concatenated with the scale information in the channel dimension and decoded through two MLP layers to obtain the second high-resolution deep features.

[0086] Step 6: After splicing the first high-resolution depth feature and the second high-resolution depth feature, send them to the fully connected layer to generate a high-resolution depth residual image; upsample the low-resolution depth image by bicubic interpolation to obtain the low-frequency component of the target depth map, add the high-resolution depth residual image and the low-frequency component of the target depth map to obtain a super-resolution reconstructed depth map of any scale.

[0087] In the above embodiment, the method is implemented by a neural network model, and the training steps corresponding to the neural network model are specifically as follows:

[0088] Get the training data and the given high-resolution depth image ground truth;

[0089] Using the training data as input, repeat steps 1 to 4 to obtain a super-resolution reconstructed depth map;

[0090] based on Regularized loss, using the error between the super-resolution reconstructed depth map and the given high-resolution depth image true value to construct the loss function, the corresponding process has the following relationship:

[0091] ;

[0092] in, represents the loss function, Respectively represent the super-resolution reconstructed depth map and the given high-resolution depth image true value, express Regular operation;

[0093] Input the loss function into the Adam optimizer for optimization, set the Adam optimizer weight decay strategy and learning parameters; minimize the loss by updating the weights and learning parameters to improve the performance of the neural network model.

[0094] Furthermore, the method for obtaining training data specifically includes the following steps:

[0095] Divide the high-resolution image dataset into training set, validation set and test set;

[0096] The high-resolution image dataset is cropped into pairs of image blocks according to a preset image size to obtain pairs of high-resolution texture images. and high-resolution depth images , and perform data augmentation through random flipping and selection operations;

[0097] Randomly select a value from (1,16] as the downsampling scale of the high-resolution depth map, and convert the high-resolution depth map Generate a low-resolution depth map of the corresponding scale through bicubic interpolation .

[0098] Please refer to Figure 2 , in the figure, Represents a splicing operation, represents local sampling;

[0099] This embodiment further provides a depth map super-resolution reconstruction system based on scale-aware implicit function, wherein the system applies the depth map super-resolution reconstruction method based on scale-aware implicit function as described above, and the system includes:

[0100] Guided feature extraction branch for:

[0101] Use convolution operation to obtain shallow guidance features of high-resolution guidance map;

[0102] Based on Transformer, a sliding window method is used to extract deep features from shallow guiding features. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guiding features.

[0103] The deep super-resolution branch is used to:

[0104] Use convolution operation to obtain shallow depth features of low-resolution depth map;

[0105] Based on Transformer, a sliding window method is used to extract deep features from shallow and deep features respectively. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain low-resolution deep features.

[0106] Bidirectional implicit fusion block for:

[0107] The high-resolution guidance features and low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain updated high-resolution guidance features and low-resolution depth features;

[0108] Encoders for:

[0109] Taking the updated high-resolution guided features and low-resolution deep features as input, the guided feature extraction branch, the deep super-resolution branch and the bidirectional implicit fusion block are iteratively repeated to obtain the final high-resolution guided features and low-resolution deep features.

[0110] Decoder for:

[0111] An external implicit Transformer based on implicit cross attention is adopted, which takes the final high-resolution guided features and low-resolution deep features as input, and uses the final high-resolution guided features to guide the final low-resolution deep features for decoding to generate the first high-resolution deep features;

[0112] An internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution deep features as input to generate the second high-resolution deep features;

[0113] After splicing the first high-resolution depth feature and the second high-resolution depth feature, they are sent to the fully connected layer to generate a high-resolution depth residual image; the low-frequency component of the target depth map obtained by bicubic interpolation upsampling the low-resolution depth image is added to the high-resolution depth residual image and the low-frequency component of the target depth map to obtain a super-resolution reconstructed depth map of any scale.

[0114] Please refer to Figure 3 , in the figure, represents matrix multiplication, Indicates addition;

[0115] Furthermore, the bidirectional implicit fusion block includes a color-guided depth refinement block and a depth-guided color selection block;

[0116] Color-guided depth refinement block for:

[0117] The low-resolution deep features are passed through a The convolutional layer gets the first query;

[0118] The high-resolution guidance features are passed through two independent The convolutional layer obtains the first key and the first value;

[0119] Establishing a mapping from a query position in low-resolution coordinates to a window center in high-resolution coordinates to determine a local window of a first key and a first value;

[0120] When mapping, select the adjacent integer position as the center of the local window, and upsample the first query according to the mapping relationship;

[0121] Performing position encoding using the relative coordinates between the first query and each element in the first key and the first value as target scale information to obtain a first scale-aware position encoding, performing implicit cross-attention calculation using the first query, the first key, and the first value, and introducing the first scale-aware position encoding in the attention calculation process to obtain a first implicit cross-attention output;

[0122] The first implicit cross-attention output is concatenated with the target scale information in the channel dimension to explicitly introduce the scale information;

[0123] The concatenated result is then fed into a feed-forward neural network consisting of two MLP layers to obtain updated low-resolution deep features.

[0124] A deeply guided color selection block for:

[0125] Upsampling the updated low-resolution depth features through bilinear interpolation to achieve the same size as the high-resolution guidance features to obtain the upsampled depth features;

[0126] The upsampled depth features are multiplied by the corresponding coordinate elements of the high-resolution guidance features and passed through an MLP layer to obtain the updated high-resolution guidance features.

[0127] The system of the present invention is composed of an encoder and a decoder. The encoder realizes the alignment of texture guidance features and depth features based on continuous coordinate mapping, and introduces implicit cross-attention to fuse the above-mentioned depth features and guidance features with continuous scale differences. This fusion process not only screens and removes irrelevant guidance features, but also refines the depth features, realizing the bidirectional fusion of guidance features and depth features. In order to improve the model's perception of the continuous reconstruction scale task, scale information is embedded in both the encoder and the decoder. Specifically, the encoder embeds scale information into the feedforward network and position encoding to generate a more refined and scale-sensitive feature representation, and the decoder realizes continuous scale depth map reconstruction based on the implicit attention mechanism embedded with scale information. In order to jointly utilize the depth map prior and the guidance prior, the decoder consists of two branches based on internal implicit self-attention and external implicit cross-attention mechanisms. The present invention provides a unified solution for super-resolution reconstruction of depth maps of arbitrary scales. At the same time, the proposed model and system are compared with the most advanced methods on mainstream data sets, and both achieve the best results of subjective evaluation and objective evaluation.

[0128] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0129] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0130] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A depth map super-resolution reconstruction method based on scale-aware implicit function, characterized in that: The method comprises the following steps: Step 1: Use convolution operation to obtain shallow guidance features of the high-resolution guidance map and shallow depth features of the low-resolution depth map; Step 2: Based on Transformer, a sliding window is used to extract deep features from shallow guide features and shallow depth features respectively. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guide features and low-resolution depth features. Step 3: The high-resolution guidance features and the low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain updated high-resolution guidance features and low-resolution depth features; Step 4: Using the updated high-resolution guide features and low-resolution depth features as input, repeat steps 1 to 3 in an iterative manner to obtain the final high-resolution guide features and low-resolution depth features; Step 5: Using an external implicit Transformer based on implicit cross attention, taking the final high-resolution guided features and low-resolution deep features as input, using the final high-resolution guided features to guide the final low-resolution deep features for decoding, and generating the first high-resolution deep features; An internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution deep features as input to generate the second high-resolution deep features; Step 6: After splicing the first high-resolution depth feature and the second high-resolution depth feature, the high-resolution depth residual image is sent to the fully connected layer to generate a high-resolution depth residual image; the low-frequency component of the target depth map obtained by bicubic interpolation upsampling the low-resolution depth image is added to the high-resolution depth residual image and the low-frequency component of the target depth map to obtain a super-resolution reconstructed depth map of any scale; The high-resolution guidance features and low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain the updated high-resolution guidance features and low-resolution depth features. The specific steps are as follows: The low-resolution deep features are passed through a The convolutional layer gets the first query; The high-resolution guidance features are passed through two independent The convolutional layer obtains the first key and the first value; Establishing a mapping from a query position in low-resolution coordinates to a window center in high-resolution coordinates to determine a local window of a first key and a first value; When mapping, select the adjacent integer position as the center of the local window, and upsample the first query according to the mapping relationship; Performing position encoding using the relative coordinates between the first query and each element in the first key and the first value as target scale information to obtain a first scale-aware position encoding, performing implicit cross-attention calculation using the first query, the first key, and the first value, and introducing the first scale-aware position encoding in the attention calculation process to obtain a first implicit cross-attention output; The first implicit cross-attention output is concatenated with the target scale information in the channel dimension to explicitly introduce the scale information; The concatenated result is then fed into a feed-forward neural network consisting of two MLP layers to obtain updated low-resolution deep features. Upsampling the updated low-resolution depth features through bilinear interpolation to achieve the same size as the high-resolution guidance features to obtain the upsampled depth features; The upsampled depth features are multiplied by the corresponding coordinate elements of the high-resolution guidance features and passed through an MLP layer to obtain the updated high-resolution guidance features.

2. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 1, characterized in that: In step 2, based on Transformer, a sliding window is used to extract deep features from shallow guide features and shallow depth features respectively; and in the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guide features and low-resolution depth features. The process specifically includes the following steps: Step 2.1, set the window size, and divide the input image into several image blocks according to the window size; Step 2.2, map the image blocks into tokens using linear transformation, and obtain the relative coordinates and scale information between any token pairs; Step 2.3, concatenate the relative coordinates and scale information, learn the scale information features through a sub-network composed of two MLP layers, and obtain the scale-aware position encoding; Step 2.4: Introduce scale-aware position encoding to perform self-attention calculation on the tokens in each window to obtain scale-aware self-attention output; Step 2.5: Repeat steps 2.1 to 2.4 using shallow guidance features and shallow depth features as input features to obtain high-resolution guidance features and low-resolution depth features.

3. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 2 is characterized in that: The process of introducing scale-aware position encoding to calculate the self-attention of the tokens in each window has the following relationship: ； in, Represent query vector, key vector, and value vector respectively; represents the transpose operation, express Q and K The number of dimensions, represents a static positional encoding of the scale-free input, represents the dynamic position encoding adjusted by the dynamic scale information, represents the self-attention calculation operation, represents the normalized exponential function.

4. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 3 is characterized in that: The first query, the first key, and the first value are used to perform implicit cross-attention calculation, and the first scale-aware position encoding is introduced in the attention calculation process to obtain the first implicit cross-attention output. The corresponding process has the following relationship: ； in, represents the implicit cross-attention computation operation, represents an element in the first query, Respectively represent the first query, the first key, and the first value; Relative coordinates. Represents a subnetwork containing three MLP layers.

5. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 4 is characterized in that: In step 5, an external implicit Transformer based on implicit cross attention is used, the final high-resolution guided feature and the low-resolution depth feature are taken as input, and the final high-resolution guided feature is used to guide the final low-resolution depth feature for decoding, and the generation of the first high-resolution depth feature specifically includes the following steps: The final low-resolution depth feature is passed through a The convolutional layer obtains the second query; The final high-resolution guided features are passed through two independent The convolutional layer obtains the second key and the second value; establishing a mapping from a query position in low-resolution coordinates to a window center in high-resolution coordinates to determine a local window for a second key and a second value; When mapping, select the adjacent integer position as the center of the local window, and upsample the second query according to the mapping relationship; Performing position encoding using the relative coordinates between the second query and each element in the second key and the second value as target scale information to obtain a second scale-aware position encoding, performing implicit cross-attention calculation using the second query, the second key, and the second value, and introducing the second scale-aware position encoding in the attention calculation process to obtain a second implicit cross-attention output; The second implicit cross-attention output is concatenated with the target scale information in the channel dimension and decoded through two MLP layers to obtain the first high-resolution deep feature.

6. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 5, characterized in that: In step 5, the internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution depth feature as input, and the generation of the second high-resolution depth feature specifically includes the following steps: The final low-resolution deep features are processed using three separate convolutional layers to generate the third query, the third key, and the third value; Upsampling the third query to the high-resolution space by bilinear interpolation to perform high-resolution coordinate traversal; The relative coordinates between the third query and each element in the third key and the third value after the above sampling, and the sine embedding is added to the relative coordinates to explicitly map the low-dimensional relative position to the high-dimensional representation. The process corresponding to the sine embedding added to the relative coordinates has the following relationship: ； in, represents the embedded output, represents a hyperparameter; The upsampled third query is used to perform self-attention calculation with the third key and the third value, and the relative coordinates after additional sine embedding are used as position encoding. The position encoding is introduced in the attention calculation process to obtain implicit self-attention output. The implicit self-attention output is concatenated with the scale information in the channel dimension and decoded through two MLP layers to obtain the second high-resolution deep features.

7. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 6, characterized in that: The method is implemented through a neural network model, and the training steps corresponding to the neural network model are as follows: Get the training data and the given high-resolution depth image ground truth; Using the training data as input, repeat steps 1 to 6 to obtain a super-resolution reconstructed depth map; based on Regularization loss uses the error between the super-resolution reconstructed depth map and the given high-resolution depth image to construct the loss function. The corresponding process has the following relationship: ； in, represents the loss function, Respectively represent the super-resolution reconstructed depth map and the given high-resolution depth image true value, express Regular operation; Input the loss function into the Adam optimizer for optimization, set the Adam optimizer weight decay strategy and learning parameters; minimize the loss by updating the weights and learning parameters to improve the performance of the neural network model.

8. The depth map super-resolution reconstruction method based on scale-aware implicit function according to claim 7, characterized in that: The method for obtaining training data specifically includes the following steps: Divide the high-resolution image dataset into training set, validation set and test set; The high-resolution image dataset is cropped into pairs of image blocks according to a preset image size to obtain pairs of high-resolution texture images. and high-resolution depth images , and perform data augmentation through random flipping and selection operations; Randomly select a value from (1,16] as the downsampling scale of the high-resolution depth map, and convert the high-resolution depth map Generate a low-resolution depth map of the corresponding scale through bicubic interpolation .

9. A depth map super-resolution reconstruction system based on scale-aware implicit function, characterized in that: The system applies the depth map super-resolution reconstruction method based on scale-aware implicit function according to any one of claims 1 to 8, and the system comprises: Guided feature extraction branch for: Use convolution operation to obtain shallow guidance features of high-resolution guidance map; Based on Transformer, a sliding window method is used to extract deep features from shallow guiding features. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain high-resolution guiding features. The deep super-resolution branch is used to: Use convolution operation to obtain shallow depth features of low-resolution depth map; Based on Transformer, a sliding window method is used to extract deep features from shallow and deep features respectively. During the extraction process, the position information and scale information of the image block obtained by the window are introduced into the self-attention calculation to learn scale perception, so as to obtain low-resolution deep features. Bidirectional implicit fusion block for: The high-resolution guidance features and low-resolution depth features are mutually fused through implicit cross attention and updated bidirectionally to obtain updated high-resolution guidance features and low-resolution depth features; Encoders for: Taking the updated high-resolution guided features and low-resolution deep features as input, the guided feature extraction branch, the deep super-resolution branch and the bidirectional implicit fusion block are iteratively repeated to obtain the final high-resolution guided features and low-resolution deep features. Decoder for: An external implicit Transformer based on implicit cross attention is adopted, which takes the final high-resolution guided features and low-resolution deep features as input, and uses the final high-resolution guided features to guide the final low-resolution deep features for decoding to generate the first high-resolution deep features; An internal implicit Transformer based on implicit self-attention is used to decode the final low-resolution deep features as input to generate the second high-resolution deep features; After splicing the first high-resolution depth feature and the second high-resolution depth feature, they are sent to the fully connected layer to generate a high-resolution depth residual image; the low-frequency component of the target depth map obtained by bicubic interpolation upsampling the low-resolution depth image is added to the high-resolution depth residual image and the low-frequency component of the target depth map to obtain a super-resolution reconstructed depth map of any scale.

Images