Unsupervised depth repair method based on depth modulation deformation Transform

Abstract

The invention relates to the technical field of computer vision and deep learning. The invention provides an unsupervised depth restoration method based on depth modulation deformation Transform, which comprises the following steps: constructing a restoration network model based on Transform, comprising a depth feature extraction module, an image feature extraction module and a depth reconstruction module; obtaining a training set, and randomly extracting sparse depth images and corresponding registration color images from the training set; obtaining a plurality of scale depth features of the sparse depth image through a depth feature extraction module, and obtaining a plurality of scale image features of the corresponding registration color image through an image feature extraction module; splicing the plurality of scale depth features and the plurality of scale image features, transmitting the spliced features to a depth reconstruction module for up-sampling scale by scale, and reconstructing to obtain a dense depth image; and establishing a corresponding loss function, and optimizing the repair network model. The problems of limited global self-attention efficiency and limited local self-attention accuracy of an existing Transform network are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to an unsupervised deep restoration method based on a deep modulation deformation Transformer. Background Art

[0002] Depth images reflect the distance between each point in the scene and the observation point, and are widely used in fields such as autonomous driving, augmented reality, and 3D reconstruction. Depth sensors such as lidar and structured light can obtain relatively accurate depth data, but due to hardware limitations, the depth images they obtain are relatively sparse. For example, the KITTI depth dataset collected by the Velodyne LiDAR HDL-64E depth sensor has only 5% effective depth pixels. The depth restoration method is a technology that can restore dense depth maps from sparse depth maps, which has important application value. Depth restoration methods can be divided into supervised depth restoration methods and unsupervised depth restoration methods. Although supervised methods can obtain better restoration results, in actual application scenarios, they often find it difficult to obtain real depth annotated images that can be used for supervised training. Therefore, unsupervised depth restoration methods have higher application flexibility, and their research and development are very necessary. In recent years, they have attracted the attention of academia and industry, and some popular methods have emerged. Ma et al. first applied deep learning to the field of unsupervised depth restoration and proposed a self-supervised sparse-to-dense restoration method (Sparse-to-dense, S2D), which achieved restoration performance far superior to traditional methods. Wong et al. considered the three-dimensional representation of RGB in depth restoration and proposed an unsupervised restoration method based on the calibrated backprojection layer (Calibrated Backprojection Network, KBNet). Yan et al. decomposed absolute depth prediction into relative depth reconstruction and absolute scale estimation, and proposed a restoration network based on decomposed scale consistency (Decomposed scale-consistent network, DesNet).

[0003] Most unsupervised deep inpainting methods rely only on the structure of convolutional neural networks. The limited perceptual domain of convolutional operations limits the global semantic feature representation capability of the inpainting network to a certain extent. Although the Transformer network has a high global semantic modeling capability, existing research still has limitations in the following two aspects:

[0004] (1) Limited global self-attention efficiency. A complete self-attention mechanism requires high computational complexity. To reduce computational complexity, methods such as Swin and HaloNets use a window-based self-attention mechanism and use operations such as moving or scaling windows to establish information interaction between windows. However, the expansion speed of its perception domain is slow. In order to achieve a larger perception domain, more Transformer blocks need to be connected in series, so the efficiency of achieving global self-attention is limited.

[0005] (2) Limited local self-attention accuracy. The existing Transformer structure does not select the token sequence, and tokens with low relevance will also participate in the self-attention operation, resulting in low local self-attention accuracy. For the field of unsupervised deep restoration, this will cause redundant texture information in color images to enter the deep restoration results, resulting in texture over-transfer defects. Summary of the invention

[0006] The purpose of the present invention is to provide an unsupervised deep restoration method based on deep modulation deformation Transformer, aiming to solve the problems of limited global self-attention efficiency and limited local self-attention accuracy of existing Transformer networks.

[0007] The present invention is achieved through the following technical solutions:

[0008] An unsupervised depth restoration method based on a depth modulation deformation Transformer comprises the following steps:

[0009] A restoration network model is constructed based on Transformer. The restoration network model includes: a deep feature extraction module, an image feature extraction module and a deep reconstruction module. The deep feature extraction module, the image feature extraction module and the deep reconstruction module are interconnected.

[0010] Obtain a training set, randomly extract sparse depth images and corresponding registered color images from the training set, and use them as iterative training inputs for the restoration network model;

[0011] A depth feature extraction module is used to obtain several scale depth features of the sparse depth image, and an image feature extraction module is used to obtain several scale image features of the corresponding registered color image;

[0012] After concatenating several scales of depth features with several scales of image features, they are passed into the depth reconstruction module for scale-by-scale upsampling, and a dense depth image is reconstructed;

[0013] Establish the corresponding loss function and optimize the repair network model.

[0014] Optionally, the deep feature extraction module includes an S2D primary network and a first convolution group, and the first convolution group is connected in series with the S2D primary network; wherein the S2D primary network is used to preliminarily repair several scale depth features, and the first convolution group is used to further extract several scale depth features.

[0015] Optionally, the image feature extraction module includes a second convolution group and an inter-window and intra-window Transformer module, the inter-window and intra-window Transformer module includes a downsampling layer and several inter-window and intra-window Transformer blocks, the downsampling layer is connected in series with the several inter-window and intra-window Transformer blocks; wherein the input of the inter-window and intra-window Transformer blocks is image features and depth features of corresponding scales.

[0016] Optionally, the inter-window and intra-window Transformer blocks include a layer normalization module, an inter-window self-attention module, an intra-window self-attention module, a weight adjustment module and a multi-layer perceptron. The inter-window self-attention module is connected in parallel with the intra-window self-attention module, and the inter-window self-attention module and the intra-window self-attention module are respectively connected in series with corresponding linear projection layers; according to the weight of the intra-window self-attention, the weight of the inter-window self-attention is adjusted by the weight adjustment module, the image features of each channel layer have the same mean and variance by the layer normalization module, and the image features and depth features of the corresponding scale are nonlinearly transformed and processed accordingly by the multi-layer perceptron.

[0017] Optionally, the window mean operation module, the linear projection module, the multiplier module, the Softmax layer module and the anti-pooling layer module calculate the window mean statistic for each image feature corresponding window through the window mean operation module; obtain the query vector, key vector and value vector of the window mean statistic through the linear projection module; multiply the query vector and the key vector through the multiplier module to obtain the attention weight, and normalize the attention weight through the Softmax layer module; multiply the normalized attention weight and the value vector through the multiplier module to perform weighted fusion between the window mean statistics; expand the weighted fused window mean statistic within the window through the anti-pooling layer module, and restore the dimension of the input image feature.

[0018] Optionally, the processing flow of the inter-window self-attention module includes:

[0019] Let the image feature after window division be X, X∈R M×N×C , where M is the number of windows, N is the number of feature tokens in each window, and C is the channel dimension of the feature token;

[0020] The mean statistics are calculated for each image feature window by performing the mean operation within the window, and the feature Then use 3 linear projections W r Q , W r K and W r V Get the query vector Q of the window statistics respectively r , key vector K r Sum value vector V r , the expressions are shown in the following formulas (1), (2) and (3):

[0021]

[0022] The inter-window self-attention is performed by taking the query vector Q r With the key vector K r The attention weight is obtained by multiplication, and the relative position code d is superimposed. The attention weight is normalized by the Softmax layer. The normalized attention weight and value vector V r Multiply them together to perform weighted fusion between window mean statistics, as shown in the following formula (4):

[0023]

[0024] Among them, Y r Represents a weighted fusion between window mean statistics.

[0025] Optionally, the in-window self-attention module includes an in-window dense graph propagation module, a depth modulation sampling module, a deformation attention module and a region stitching module, and the in-window dense graph propagation module, the depth modulation sampling module, the deformation attention module and the region stitching module are connected in series in sequence; wherein the in-window dense graph propagation module is used to eliminate noise in the depth feature; the depth modulation sampling module is used to perform channel compression and back-projection of the depth feature from two-dimensional pixel points to three-dimensional pixel point cloud, obtain the normal and tangent plane corresponding to the center point in the window by the normal estimation method, preset a judgment threshold, calculate the distance value from the remaining domain points in the window to the tangent plane corresponding to the center point, and the domain points with a distance value greater than the judgment threshold are judged to be in the same target as the center point, forming a related area, and the remaining domain points constitute a non-related area; the deformation attention module is used to perform weighted fusion between image features in the related area, update the related area through self-attention, and perform corresponding projection on the non-related area; the region stitching module is used to stitch the related area after the self-attention update and the non-related area after the corresponding projection into a regular window feature.

[0026] Optionally, the specific process of obtaining several scale depth features of the sparse depth image through the depth feature extraction module and obtaining several scale image features of the corresponding registered color image through the image feature extraction module is:

[0027] Let the training set be in, represents the i-th sparse depth image, represents the i-th corresponding registered color image, from the training set Randomly extract several sparse depth images and several corresponding registered color images and input them into the restoration network;

[0028] The sparse depth image is input into the depth feature extraction module to obtain multi-scale depth features in turn. The expressions are shown in the following equations (5) and (6):

[0029]

[0030]

[0031] in, represents the shallow depth features extracted by the S2D primary network and the first two convolutional layers, f S2D and Denote the functional functions corresponding to the S2D primary network and the first two convolutional layers, respectively. t represents a sparse depth image, represents the depth feature of the s-th scale, Represents the functional function of the s-th scale convolutional layer;

[0032] The corresponding registered color image is input into the image feature extraction module to obtain multi-scale image features in turn. The expressions are shown in the following equations (7) and (8):

[0033]

[0034]

[0035] in, represents the shallow image features extracted by the first two convolutional layers, Represents the function of shallow image feature extraction, represents the image features extracted by the inter-window and intra-window Transformer groups of the s-th scale, Represents the function of the Transformer group between windows and within windows at the s-th scale, through the image features at the s-1th scale And the deep features of the sth scale passed by the deep feature extraction module To learn the image features of the s-th scale

[0036] Optionally, the specific process of concatenating the depth features of several scales with the image features of several scales, passing them into the depth reconstruction module for upsampling scale by scale, and reconstructing a dense depth image is as follows:

[0037] The image features of the s-th scale and deep features After splicing, it is passed to the depth reconstruction module for scale-by-scale upsampling and reconstructing a dense depth image, as shown in the following equations (9) and (10):

[0038]

[0039]

[0040] Among them, ∥ represents the feature concatenation operation, represents the deep reconstruction feature of the s-th scale, Represents the deconvolution layer function corresponding to the s+1th scale of the deep reconstruction module, Represents the convolutional layer function corresponding to the s+1th scale of the deep reconstruction module, represents the final reconstructed dense depth image, f IIWTNet Represents the function of repairing the network model.

[0041] Optionally, the specific process of establishing a corresponding loss function and optimizing the repair network model is:

[0042] Through the current frame image I t With adjacent frame I s The geometric constraints between them, combined with the pose network N P The estimated camera transformation matrix T between the previous and next frames t→s and the reconstructed dense depth image Get the reprojected image of the current frame By constructing the current frame image I t With the reprojected image The photometric error between them is used to complete the training of the restoration network model, where the generation method of the reprojected image is expressed as follows:

[0043]

[0044] in, Represents the homogeneous coordinates of the reconstructed image point p, K is the camera internal parameter, and π is the homogeneous coordinates Transform to the non-homogeneous coordinates p in the image plane s , if p s If it is a non-integer coordinate, then i s Linear interpolation is performed as the value of point p in the reconstructed image. The photometric error measures the current frame image I t With the reprojected image The L1 difference, SSIM difference and feature difference between them are expressed as follows (12):

[0045]

[0046] Among them, L ph represents the optical error, SSIm represents the SSIM difference function, and Z t Represent the autoencoder reconstruction network N R From I t and The extracted coding features, self-encoding reconstruction network N R The error function can be incorporated into the total error, α, β and γ represent the coefficients corresponding to L1 difference, SSIM difference and feature difference respectively;

[0047] Sparse depth error and smooth error are also used in training. The sparse depth error measures the reconstructed dense depth image. With the sparse depth image D t In D t The difference at the effective point is expressed as follows (13):

[0048]

[0049] Among them, L sd represents the sparse depth error, Ψ represents the sparse depth image D t The effective point set of the smoothing error is to use the image gradient to adjust the error weight of the depth gradient, and the expression is shown in the following formula (14):

[0050]

[0051] Among them, L sm represents the smoothing error; the total loss function L(θ) combines the photometric error, sparse depth error and smoothing error, and its expression is shown in the following formula (15):

[0052] L(θ)=L ph +λ sd L sd +λ sm L sm (15)

[0053] Among them, Θ represents the parameters of the repair network model, λ sd and λ sm are the regularization parameters for the sparse depth error and smooth error, respectively.

[0054] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0055] Improve the quality of deep image restoration: By constructing a Transformer-based restoration network model, it is possible to efficiently fuse the information of sparse depth images and corresponding registered color images, and use the deep feature extraction module and image feature extraction module to capture the multi-scale features of the two respectively, and then use the deep reconstruction module to achieve accurate and delicate depth image reconstruction; compared with traditional methods, it can significantly improve the restoration quality of depth images, making the reconstructed depth images closer to the real scene.

[0056] Enhanced applicability of unsupervised learning: An unsupervised deep inpainting strategy is adopted, which does not rely on a large amount of labeled data and only requires the correspondence between sparse depth images and registered color images for training, which greatly reduces the cost of data preparation and enhances the wide applicability of this method in practical applications, especially when it is difficult to obtain high-quality labeled data.

[0057] Realize effective fusion of multi-scale features: Through the deep feature extraction module and the image feature extraction module, several scale features of the sparse depth image and the color image are extracted respectively, and these features are spliced ​​to achieve cross-modal and multi-scale feature fusion, which helps to capture the spatial structure and detail information at different scales, and provides a rich information basis for the accurate reconstruction of the depth image.

[0058] Optimize network model performance: By establishing the corresponding loss function and optimizing the repair network model, the reconstruction ability of the model can be continuously improved, making it better adaptable to different types of input data, ensuring the stability and reliability of the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of an unsupervised depth restoration method based on a depth modulation deformation Transformer according to an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of a process flow for repairing a network model in Embodiment 2 of the present invention;

[0061] Figure 3 Schematic diagram of the processing flow of the inter-window and intra-window Transformer blocks of the repair network model in Embodiment 2 of the present invention;

[0062] Figure 4 Schematic diagram of the processing flow of the inter-window self-attention module of the repair network model in Example 2 of the present invention;

[0063] Figure 5 Schematic diagram of the processing flow of the in-window self-attention module of the repair network model in Example 2 of the present invention;

[0064] Figure 6This is a comparison table of objective indicator values ​​of Example 3 of the present invention on the KITTI test set.

[0065] Legend: 1000-image feature extraction module, 1100-inter-window and intra-window Transformer modules, 1110-inter-window self-attention module, 1120-intra-window self-attention module, 1121-intra-window dense graph propagation module, 1122-depth modulation sampling module, 1123-deformation attention module, 2000-depth feature extraction module, 3000-depth reconstruction module. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0067] Example 1

[0068] Reference Figure 1 , Figure 2 , an unsupervised deep restoration method based on deep modulation deformation Transformer, comprising the steps of:

[0069] Step 1: construct a restoration network model based on Transformer, the restoration network model includes: a depth feature extraction module 2000, an image feature extraction module 1000 and a depth reconstruction module 3000, and the depth feature extraction module 2000, the image feature extraction module 1000 and the depth reconstruction module 3000 are interconnected;

[0070] Step 2: Obtain a training set, randomly extract sparse depth images and corresponding registered color images from the training set, and use them as iterative training inputs for the restoration network model;

[0071] Step 3: obtaining several scale depth features of the sparse depth image through the depth feature extraction module 2000, and obtaining several scale image features of the corresponding registered color image through the image feature extraction module 1000;

[0072] Step 4: After concatenating the depth features of several scales with the image features of several scales, the features are transmitted to the depth reconstruction module 3000 for upsampling scale by scale, and a dense depth image is reconstructed;

[0073] Step 5: Establish the corresponding loss function and optimize the repair network model.

[0074] In this embodiment, Figure 2As shown, the deep feature extraction module 2000 includes an S2D primary network and a first convolution group, and the first convolution group is connected in series with the S2D primary network; wherein the S2D primary network is used to preliminarily repair several scale depth features, and the first convolution group is used to further extract several scale depth features.

[0075] In this embodiment, Figure 2 As shown, the image feature extraction module includes a second convolution group and an inter-window and intra-window Transformer module, the inter-window and intra-window Transformer module includes a downsampling layer and a plurality of inter-window and intra-window Transformer blocks, the downsampling layer is connected in series with the plurality of inter-window and intra-window Transformer blocks; wherein the input of the inter-window and intra-window Transformer blocks is the image features and depth features of the corresponding scale.

[0076] In this embodiment, Figure 2 As shown, the image feature extraction module 1000 includes a second convolution group and four inter-window and intra-window Transformer modules, wherein the second convolution group includes a convolution layer with a step size of 1 and a convolution layer with a step size of 2, and each inter-window and intra-window Transformer module includes a downsampling layer and two serially connected inter-window and intra-window Transformer blocks, and the multi-scale RGB image features are extracted through the image feature extraction module 1000; the deep feature extraction module 2000 includes an S2D primary network and a first convolution group, and the first convolution group includes 1 convolution layer with a stride of 1 and 5 convolution layers with a stride of 2, the S2D primary network is used to preliminarily repair the depth features, and the subsequent multiple convolution layers in series are used to extract multi-scale depth features; the depth reconstruction module 3000 includes multiple deconvolution layers (DeConv) and convolution layers (Conv) in series, the deconvolution layer is used to upsample the decoding features, and the convolution layer integrates the multi-scale image features and multi-scale depth features transmitted by the image feature extraction module 1000 and the depth feature extraction module 2000 into the decoding features, and finally outputs a dense depth repaired image.

[0077] Example 2

[0078] Based on Example 1, refer to Figure 3 , Figure 4 , Figure 5In this embodiment, the inter-window and intra-window Transformer blocks include a layer normalization module, an inter-window self-attention module 1110, an intra-window self-attention module 1120, a weight adjustment module and a multi-layer perceptron. The inter-window self-attention module 1110 is connected in parallel with the intra-window self-attention module 1120, and the inter-window self-attention module 1110 and the intra-window self-attention module 1120 are respectively connected in series with corresponding linear projection layers; according to the weight of the intra-window self-attention, the weight of the inter-window self-attention is adjusted by the weight adjustment module, the image features of each channel layer have the same mean and variance by the layer normalization module, and the image features and depth features of the corresponding scale are nonlinearly transformed and processed accordingly by the multi-layer perceptron.

[0079] In this embodiment, Figure 3 As shown, the inter-window and intra-window Transformer module 1100 includes two layer normalization modules, one inter-window self-attention module 1110, one intra-window self-attention module 1120, one weight adjustment module and one multi-layer perceptron (MLP). The input of the inter-window and intra-window Transformer module 1100 is the image features and the depth features transmitted by the depth feature extraction module 2000 at the corresponding scale. The depth features mainly play a role in the intra-window self-attention module 1120. The depth features generate the distance from the three-dimensional neighborhood point to the center point tangent plane, referred to as the point-to-plane distance P2P, which guides the distinction between relevant and irrelevant areas in the image feature window, and limits the self-attention operation to the relevant area, thereby avoiding the image feature token with low correlation from participating in the self-attention operation, and improving the accuracy of local self-attention. Layer normalization is to make the image features of each channel layer have the same mean and variance, so as to achieve a stable training model, while realizing effective global dependency learning and accurate local feature learning.

[0080] In this embodiment, Figure 4 As shown, the inter-window self-attention module 1110 includes 1 intra-window mean operation module, 3 linear projection modules, 2 multiplier modules, 1 Softmax layer module and 1 anti-pooling layer module. The window mean statistic is calculated for each image feature corresponding window through the intra-window mean operation module; the query vector, key vector and value vector of the window mean statistic are respectively obtained through the three linear projection modules; the query vector and the key vector are multiplied by one of the multiplier modules to obtain the attention weight, and the attention weight is normalized by the Softmax layer module; the normalized attention weight and the value vector are multiplied by the multiplier module to perform weighted fusion between the window mean statistics; the weighted fused window mean statistic is expanded within the window through the anti-pooling layer module, and the dimension of the input image feature is restored.

[0081] In this embodiment, Figure 4As shown, the processing flow of the inter-window self-attention module 1110 includes:

[0082] Let the image feature after window division be X, X∈R M×N×C , where M is the number of windows, N is the number of feature tokens in each window, and C is the channel dimension of the feature token;

[0083] The window mean operation module is used to perform the window mean operation, and the mean statistics are calculated for each image feature window. Then use 3 linear projection modules W r Q , W r K and W r V Get the query vector Q of the window statistics respectively r , key vector K r Sum value vector V r , the expressions are shown in the following formulas (1), (2) and (3):

[0084]

[0085]

[0086] The inter-window self-attention is performed by taking the query vector Q r With the key vector K r The attention weight is obtained by multiplication, and the relative position code d is superimposed. The attention weight is normalized by the Softmax layer of the Softmax layer module. The normalized attention weight and value vector V r The weighted fusion between window mean statistics is performed by multiplying them through the multiplier module, and the expression is shown in the following formula (4):

[0087]

[0088] Among them, Y r Represents a weighted fusion between window mean statistics.

[0089] In this embodiment, Figure 5As shown, the in-window self-attention module 1120 includes an in-window dense graph propagation module 1121, a depth modulation sampling module 1122, a deformation attention module 1123 and a region splicing module, and the in-window dense graph propagation module 1121, the depth modulation sampling module 1122, the deformation attention module 1123 and the region splicing module are connected in series in sequence; wherein, the input of the in-window dense graph propagation module 1121 is the depth feature transmitted from the corresponding scale of the deep feature extraction module 2000, which includes channel compression, adjacency matrix calculation and graph propagation based on the adjacency matrix, and the in-window dense graph propagation module 1121 is used to eliminate noise in the deep feature; the depth modulation sampling module 1122 The input of module 1122 is the image features and the depth features updated by graph propagation, including channel compression, two-dimensional (2D) to three-dimensional (3D) back projection, normal estimation, P2P distance calculation and feature sampling in the image window based on P2P distance. The depth modulation sampling module 1122 is used to perform channel compression and back projection of two-dimensional pixel points to three-dimensional pixel point cloud on the depth features, obtain the normal and tangent plane corresponding to the center point in the window by the normal estimation method, preset the judgment threshold, calculate the distance value of the remaining domain points in the window to the tangent plane corresponding to the center point, and the domain points with a distance value greater than the judgment threshold are judged to be in the same target as the center point, forming a related area (Ω + ), the remaining area points constitute the non-correlated area (Ω - ); The deformation attention module 1123 includes 3 linear projection modules, 2 multiplier modules and 1 Softmax layer module. The deformation attention module is used to perform weighted fusion between image features in the relevant area, update the relevant area through self-attention, and perform corresponding projection on the non-relevant area; the three linear projection modules W a Q , W a K and W a V Get any shape related area (Ω + ), the query vector, key vector and value vector of the image features in the image, the query vector and key vector are input into a multiplier module to obtain the attention weight, the Softmax layer module is responsible for normalizing the attention weight, and another multiplier module is used for the normalized attention weight and the related area of ​​the value vector (Ω + ), weighted fusion between image features; for non-correlated areas (Ω - ), in order to ensure that the relevant area (Ω + ) features are in the same projection space and are only passed through the linear projection module W a V Linear projection is performed without participating in self-attention operation. The region splicing module is responsible for updating the relevant region (Ω + ) and the non-correlated area after projection (Ω -) are spliced ​​into regular window features for subsequent processing.

[0090] In this embodiment, the specific process of obtaining several scale depth features of the sparse depth image through the depth feature extraction module 2000 and obtaining several scale image features of the corresponding registered color image through the image feature extraction module 1000 is as follows:

[0091] Let the training set be in, represents the i-th sparse depth image, represents the i-th corresponding registered color image, from the training set Randomly extract several sparse depth images and several corresponding registered color images and input them into the restoration network;

[0092] The sparse depth image is input into the depth feature extraction module 2000 to obtain multi-scale depth features in sequence, as shown in the following equations (5) and (6):

[0093]

[0094] in, represents the shallow depth features extracted by the S2D primary network and the first two convolutional layers, f S2D and Denote the functional functions corresponding to the S2D primary network and the first two convolutional layers, respectively. t represents a sparse depth image, represents the depth feature of the s-th scale, Represents the functional function of the s-th scale convolutional layer;

[0095] The corresponding registered color image is input into the image feature extraction module 1000 to obtain multi-scale image features in sequence, as shown in the following equations (7) and (8):

[0096]

[0097]

[0098] in, represents the shallow image features extracted by the first two convolutional layers, Represents the function of shallow image feature extraction, represents the image features extracted by the inter-window and intra-window Transformer modules 1100 at the s-th scale, represents the function of the Transformer module 1100 between windows and within windows at the s-th scale, through the image features at the s-1th scale and the depth feature of the sth scale transmitted by the depth feature extraction module 2000 To learn the image features of the s-th scale

[0099] In this embodiment, after the multiple scale depth features and the multiple scale image features are spliced, they are transmitted to the depth reconstruction module 3000 for scale-by-scale upsampling, and the specific process of reconstructing a dense depth image is as follows:

[0100] The image features of the s-th scale and deep features After splicing, the images are passed to the depth reconstruction module 3000 for scale-by-scale upsampling and reconstructing a dense depth image, as shown in the following equations (9) and (10):

[0101]

[0102]

[0103] Among them, ∥ represents the feature concatenation operation, represents the deep reconstruction feature of the s-th scale, represents the deconvolution layer function corresponding to the s+1th scale of the depth reconstruction module 3000, represents the convolutional layer function corresponding to the s+1th scale of the deep reconstruction module 3000, represents the final reconstructed dense depth image, f IIWTNrt Represents the function of repairing the network model.

[0104] In this embodiment, the specific process of establishing the corresponding loss function and optimizing the repair network model is as follows:

[0105] Through the current frame image I t With adjacent frame I s The geometric constraints between them, combined with the pose network N P The estimated camera transformation matrix T between the previous and next frames t→s and the reconstructed dense depth image Get the reprojected image of the current frame By constructing the current frame image I t With the reprojected image The photometric error between them is used to complete the training of the restoration network model, where the generation method of the reprojected image is expressed as follows:

[0106]

[0107] in, Represents the homogeneous coordinates of the reconstructed image point p, K is the camera internal parameter, and π is the homogeneous coordinates Transform to the non-homogeneous coordinates p in the image planes , if p s If it is a non-integer coordinate, then I s Linear interpolation is performed as the value of point p in the reconstructed image. The photometric error measures the current frame image I t With the reprojected image The L1 difference, SSIM difference and feature difference between them are expressed as follows (12):

[0108]

[0109] Among them, L p h represents the optical error, SSIM represents the SSIM difference function, and Z t Represent the autoencoder reconstruction network N R From I t and The extracted coding features, self-encoding reconstruction network N R The error function can be incorporated into the total error, α, β and γ represent the coefficients corresponding to the L1 difference, SSIM difference and feature difference respectively. In this embodiment, α=0.15, β=0.95, γ=0.001;

[0110] Sparse depth error and smooth error are also used in training. The sparse depth error measures the reconstructed dense depth image. With the sparse depth image D y In D t The difference at the effective point is expressed as follows (13):

[0111]

[0112] Among them, L sd represents the sparse depth error, Ψ represents the sparse depth image D t The effective point set of the smoothing error is to use the image gradient to adjust the error weight of the depth gradient, and the expression is shown in the following formula (14):

[0113]

[0114] Among them, L sm represents the smoothing error; the total loss function L(θ) combines the photometric error, sparse depth error and smoothing error, and its expression is shown in the following formula (15):

[0115] L(θ)=L p h+λ sd L sd +λ sm L dm (15)

[0116] Among them, Θ represents the parameters of the repair network model, λ sd and λ sm are the regularization parameters for the sparse depth error and smooth error, respectively.

[0117] In this embodiment, the purpose of the in-window dense image propagation module 1121 is to eliminate the noise in the depth feature and provide clean and accurate geometric structure information for the subsequent depth modulation sampling module 1122. First, the depth feature F D Perform channel compression to obtain the estimated depth d(p), and then calculate Back-project the two-dimensional pixel point p into a three-dimensional point P; let the set of N pixels in the given window w be Π={p1,p2,…,p N}, G(V,E) is the graph defined on the window w, where V is the node corresponding to the set Π, and E∈|V|×|V| is the set of edges. The adjacency matrix A∈R of the dense graph G is N×N The definition is shown in the following formula (16):

[0118]

[0119] Among them, a i,j It is an element in the adjacency matrix A, which represents the correlation between node i and node j. Δp i,j =P i -P j ,in, and Δp i,j They represent the depth feature difference and three-dimensional space distance between node i and node j respectively. A is the adjacency matrix generation function, which is implemented by a 2-layer MLP. The adjacency matrix is ​​normalized by the Softmax layer and passed into the graph update step. The deep feature update of node i is obtained by weighted combination of adjacent nodes j. The expression is shown in the following formula (17):

[0120]

[0121] in, represents the updated depth feature after graph propagation; the updated depth feature is passed to the subsequent depth modulation sampling module 1122, the input of the depth modulation sampling module 1122 is the image feature and the updated depth feature after graph propagation, first the updated depth feature after graph propagation Channel compression is performed to obtain the estimated depth, and then the two-dimensional pixel points are back-projected into three-dimensional points. The normal vector n at the center point P of the window is obtained according to the surface normal estimation method. P =[n x ,n y ,n z]. Once the three-dimensional point P and its normal vector n P Determine, then the corresponding tangent plane Γ P Also determine that the tangent plane Γ P The expression of is shown in formula (18):

[0122] P T n P +φ P =0(18)

[0123] Among them, φ P It means starting from the origin of the coordinate system, along the normal line n P The distance from the direction to the tangent plane. For the neighboring point Q of point P, there is a point Q passing through the tangent plane Γ P Parallel plane Γ Q , plane Γ Q The expression of is shown in formula (19):

[0124] Q T n P +φ Q =0(19)

[0125] Then the tangent plane Γ from the neighborhood point in the window of P to P P The distance between (referred to as point-to-plane distance P2P) is expressed as follows:

[0126] Δφ Q =|φ Q -φ P |=|(QP) T n P | (20)

[0127] The median of the maximum P2P distance and the minimum P2P distance in the window is selected as the threshold, and the area greater than the threshold is determined as the relevant area (Ω + ), the area smaller than the threshold is judged as a non-correlated area (Ω - ). The deformation attention module 1123 performs the deformation attention on the relevant area (Ω + ) and non-correlated regions (Ω - ) image features are processed in different ways. + ) image features, using 3 linear projections D a Q , W a K and W a V Get the query vector Q of the window statistics respectively a , key vector K a Sum value vector V a , the self-attention in the window is calculated by taking the query vector Q aWith the key vector K a The attention weights are obtained by multiplication, and the attention weights are normalized by the Softmax layer. The normalized attention weights and the value vector V a Multiply to achieve the relevant area (Ω + ) weighted fusion of image features; for non-correlated areas (Ω - ) image features, in order to ensure that they are in the same projection space as the related area features, only linear projection W is performed a V , no self-attention operation is performed. Region stitching is responsible for stitching the relevant region after self-attention update and the non-relevant region after projection into regular window features.

[0128] Example 3

[0129] Based on Example 1 and Example 2, in this embodiment, 86K samples in KITTI are used as the training set, the test set is the KITTI test set, the KITTI test set contains 1000 samples officially provided, and the evaluation of the test set is submitted to the Internet for online evaluation.

[0130] In this embodiment, the model is trained and tested on the pytorch framework. The sparse depth images and corresponding registered color images in the training set are cropped to a size of 768×320, and 8 samples are randomly selected for training in each iteration. The Adam gradient descent method is used to optimize the network parameter Θ, where the parameters of the Adam optimizer are set to β1=0.9 and β2=0.999. The initial learning rate is set to 0.00005, the total training epoch is 60, and the learning rate is adjusted to 0.0001, 0.00015, 0.0001, 0.00005, and 0.00002 when the epochs are 2, 8, 20, 30, and 45, respectively. The regularization parameters of the sparse depth error and the smoothing error in the error function are set to 0.6 and 0.04, respectively. The root mean square error (RMSE), mean absolute error (MAE), inverse root mean square error (iRMSE), and inverse mean absolute error (iMAE) are used to evaluate the model performance.

[0131] In order to test the performance of the model, 11 representative supervised / unsupervised deep restoration methods were selected for comparison with the experimental results of the present invention. Figure 6 As shown, IIWTNet is the method proposed by the present invention. 11 representative deep repair methods include:

[0132] Method 1 (S2D): Unsupervised depth inpainting method proposed by Ma et al., reference “F. Ma, G. V. Cavalheiro, and S. Karaman, Self-supervised sparse-to-dense: Self-supervised depth completion from Iidar ​​and monocular camera, in Proc. International Conference on Robotics and Automation, 2019, pp. 3288-3295.”

[0133] Method 2 (IP-Basic): Unsupervised deep inpainting method proposed by Ku et al., reference “J. Ku, A. Harakeh, and SL Waslander, In defense of classical image processing: Fast depth completion on the cpu, in Proc. 15th Conference on Computer and Robot Vision, 2018, pp. 16-22.”.

[0134] Method 3 (DFuseNet): Unsupervised deep inpainting method proposed by Shivakumar et al., reference “SS Shivakumar, T. Nguyen, IDMiller, SW Chen, V. Kumar, and CJ Taylor, DFusenet: Deep fusion of rgb and sparse depth information for image guided dense depth completion, in Proc. IEEE Intelligent Transportation Systems Conference, 2019, pp. 13-20.”.

[0135] Method 4 (DDP): Unsupervised deep inpainting method proposed by Yang et al., reference “Y. Yang, A. Wong, and S. Soatto, Dense depth posterior (ddp) from single image and sparse range, in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3353-3362.”.

[0136] Method 5 (VOICED): Unsupervised depth completion method proposed by Wong et al., reference “A. Wong, X. Fei, S. Tsuei, and S. Soatto, Unsupervised depth completion from visual inertialodometry, IEEE Robotics and Automation Letters 5(2) (2020) pp. 1899-1906.”.

[0137] Method 6 (AdaFrame): Unsupervised depth completion method proposed by Wong et al., reference “A.Wong, X.Fei, BWHong, and S.Soatto, An adaptive framework for learning unsupervised depth completion, IEEE Robotics and Automation Letters 6(2)(2021)pp.3120-3127.”.

[0138] Method 7 (ScaffNet): Unsupervised depth completion method proposed by Wong et al., reference “A. Wong, S. Cicek, and S. Soatto, Learning topology from synthetic data for unsupervised depth completion, IEEE Robotics and Automation Letters 6(2)(2021)pp.1495-1502.”.

[0139] Method 8 (SynthProj): Unsupervised depth completion method proposed by Lopez-Rodriguez et al., reference “A. Lopez-Rodriguez, B. Busam, and K. Mikolajczyk, Project to adapt: ​​Domain adaptation for depth completion from noisy and sparse sensor data, in Proc. Asian Conference on Computer Vision, 2020.”.

[0140] Method 9 (CSPN): Supervised deep inpainting method proposed by Cheng et al., reference “X. Cheng, P. Wang, and R. Yang, Depth estimation via affinity learned with convolutional spatial propagation network, in Proc. European Conference on Computer Vision, 2018, pp. 103-119.”.

[0141] Method 10 (KBNet): Unsupervised depth completion method proposed by Wong et al., reference “A. Wong, and S. Soatto, Unsupervised depth completion with calibrated backprojection layers, in Proc. IEEE International Conference on Computer Vision, 2021, pp. 12747-12756.”

[0142] Method 11 (DesNet): Unsupervised depth completion method proposed by Yan et al., reference “Z. Yan, K. Wang, X. Li, Z. Zhang, J. Li, and J. Yang, Desnet: Decomposed scale-consistent network for unsupervised depth completion, in Proc. AAAI Conference on Artificial Intelligence, 2023, pp. 3109-3117.”.

[0143] Figure 6In the example, the best value and the second best value are indicated by black bold and underline respectively. Figure 6 It can be seen that in most cases, the four evaluation indicators of the present invention are the highest, and the reconstruction and repair effect is significantly better than some currently representative unsupervised deep repair methods.

[0144] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An unsupervised depth restoration method based on deep modulation deformation Transformer, characterized in that: Includes steps: A restoration network model is constructed based on Transformer. The restoration network model includes: a deep feature extraction module, an image feature extraction module and a deep reconstruction module. The deep feature extraction module, the image feature extraction module and the deep reconstruction module are interconnected. Obtain a training set, randomly extract sparse depth images and corresponding registered color images from the training set, and use them as iterative training inputs for the restoration network model; A depth feature extraction module is used to obtain several scale depth features of the sparse depth image, and an image feature extraction module is used to obtain several scale image features of the corresponding registered color image; After concatenating several scales of depth features with several scales of image features, they are passed into the depth reconstruction module for scale-by-scale upsampling, and a dense depth image is reconstructed; Establish the corresponding loss function and optimize the repair network model.

2. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 1, characterized in that: The deep feature extraction module includes an S2D primary network and a first convolution group, and the first convolution group is connected in series with the S2D primary network; wherein the S2D primary network is used to preliminarily repair a plurality of scale depth features, and the first convolution group is used to further extract a plurality of scale depth features.

3. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 1, characterized in that: The image feature extraction module includes a second convolution group and an inter-window and intra-window Transformer module, the inter-window and intra-window Transformer module includes a downsampling layer and a plurality of inter-window and intra-window Transformer blocks, the downsampling layer and the plurality of inter-window and intra-window Transformer blocks are connected in series; wherein the input of the inter-window and intra-window Transformer blocks is the image features and depth features of the corresponding scale.

4. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 3, characterized in that: The inter-window and intra-window Transformer blocks include a layer normalization module, an inter-window self-attention module, an intra-window self-attention module, a weight adjustment module and a multi-layer perceptron. The inter-window self-attention module is connected in parallel with the intra-window self-attention module, and the inter-window self-attention module and the intra-window self-attention module are respectively connected in series with corresponding linear projection layers; according to the weight of the intra-window self-attention, the weight of the inter-window self-attention is adjusted by the weight adjustment module, the image features of each channel layer have the same mean and variance by the layer normalization module, and the image features and depth features of corresponding scales are nonlinearly transformed and processed accordingly by the multi-layer perceptron.

5. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 4, characterized in that: The inter-window self-attention module includes an intra-window mean operation module, a linear projection module, a multiplier module, a Softmax layer module and an anti-pooling layer module. The window mean statistic is calculated for each image feature corresponding window through the intra-window mean operation module; the query vector, key vector and value vector of the window mean statistic are obtained through the linear projection module; the query vector and the key vector are multiplied by the multiplier module to obtain the attention weight, and the attention weight is normalized by the Softmax layer module; the normalized attention weight and the value vector are multiplied by the multiplier module to perform weighted fusion between the window mean statistics; the weighted fused window mean statistic is expanded within the window through the anti-pooling layer module, and the dimension of the input image feature is restored.

6. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 5, characterized in that: The processing flow of the inter-window self-attention module includes: Let the image feature after window division be X, X∈R M×N×C , where M is the number of windows, N is the number of feature tokens in each window, and C is the channel dimension of the feature token; The mean statistics are calculated for each image feature window by performing the mean operation within the window, and the feature Use 3 more linear projections and Get the query vector Q of the window statistics respectively r , key vector K r Sum value vector V r , the expressions are shown in the following formulas (1), (2) and (3): The inter-window self-attention is performed by taking the query vector Q r With the key vector K r The attention weight is obtained by multiplication, and the relative position code d is superimposed. The attention weight is normalized by the Softmax layer. The normalized attention weight and value vector V r Multiply them together to perform weighted fusion between window mean statistics, as shown in the following formula (4): Among them, Y r Represents a weighted fusion between window mean statistics.

7. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 4, characterized in that: The in-window self-attention module includes an in-window dense graph propagation module, a depth modulation sampling module, a deformation attention module and a region stitching module, and the in-window dense graph propagation module, the depth modulation sampling module, the deformation attention module and the region stitching module are connected in series in sequence; wherein the in-window dense graph propagation module is used to eliminate noise in the depth feature; the depth modulation sampling module is used to perform channel compression on the depth feature and back-project the two-dimensional pixel point to the three-dimensional pixel point cloud, obtain the normal and tangent plane corresponding to the center point in the window by the normal estimation method, preset a judgment threshold, calculate the distance value from the remaining domain points in the window to the tangent plane corresponding to the center point, and the domain points with a distance value greater than the judgment threshold are judged to be in the same target as the center point, forming a related area, and the remaining domain points constitute a non-related area; the deformation attention module is used to perform weighted fusion between image features in the related area, update the related area through self-attention, and perform corresponding projection on the non-related area; the region stitching module is used to stitch the related area after the self-attention update and the non-related area after the corresponding projection into a regular window feature.

8. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 1, characterized in that: The specific process of obtaining several scale depth features of the sparse depth image through the depth feature extraction module and obtaining several scale image features of the corresponding registered color image through the image feature extraction module is as follows: Let the training set be in, represents the i-th sparse depth image, represents the i-th corresponding registered color image, from the training set Randomly extract several sparse depth images and several corresponding registered color images and input them into the restoration network; The sparse depth image is input into the depth feature extraction module to obtain multi-scale depth features in turn. The expressions are shown in the following equations (5) and (6): in, represents the shallow depth features extracted by the S2D primary network and the first two convolutional layers, f S2D and Denote the functional functions corresponding to the S2D primary network and the first two convolutional layers, respectively. t represents a sparse depth image, represents the depth feature of the s-th scale, Represents the functional function of the s-th scale convolutional layer; The corresponding registered color image is input into the image feature extraction module to obtain multi-scale image features in turn. The expressions are shown in the following equations (7) and (8): in, represents the shallow image features extracted by the first two convolutional layers, Represents the function of shallow image feature extraction, represents the image features extracted by the inter-window and intra-window Transformer groups of the s-th scale, Represents the function of the Transformer group between windows and within windows at the s-th scale, through the image features at the s-1th scale And the deep features of the sth scale passed by the deep feature extraction module To learn the image features of the s-th scale 9. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 8, characterized in that: The specific process of concatenating the depth features of several scales with the image features of several scales, passing them into the depth reconstruction module for upsampling scale by scale, and reconstructing a dense depth image is as follows: The image features of the s-th scale and deep features After splicing, it is passed to the depth reconstruction module for scale-by-scale upsampling and reconstructing a dense depth image, as shown in the following equations (9) and (10): Among them, ∥ represents the feature concatenation operation, represents the deep reconstruction feature of the s-th scale, Represents the deconvolution layer function corresponding to the s+1th scale of the deep reconstruction module, Represents the convolutional layer function corresponding to the s+1th scale of the deep reconstruction module, represents the final reconstructed dense depth image, f IIWTNet Represents the function of repairing the network model.

10. The unsupervised depth restoration method based on depth modulation deformation Transformer according to claim 9, characterized in that: The specific process of establishing the corresponding loss function and optimizing the repair network model is as follows: Through the current frame image I t With adjacent frame I s The geometric constraints between them, combined with the pose network N P The estimated camera transformation matrix T between the previous and next frames t→s and the reconstructed dense depth image Get the reprojected image of the current frame By constructing the current frame image I t With the reprojected image The photometric error between them is used to complete the training of the restoration network model, where the generation method of the reprojected image is expressed as follows: in, Represents the homogeneous coordinates of the reconstructed image point p, K is the camera internal parameter, and π is the homogeneous coordinates Transform to the non-homogeneous coordinates p in the image plane s , if p s If it is a non-integer coordinate, then I s Linear interpolation is performed as the value of point p in the reconstructed image. The photometric error measures the current frame image I t With the reprojected image The L1 difference, SSIM difference and feature difference between them are expressed as follows (12): Among them, L ph represents the optical error, SSIM represents the SSIM difference function, and Z t Represent the autoencoder reconstruction network N R From I t and The extracted coding features, self-encoding reconstruction network N R The error function can be incorporated into the total error, α, β and γ represent the coefficients corresponding to L1 difference, SSIM difference and feature difference respectively; Sparse depth error and smooth error are also used in training. The sparse depth error measures the reconstructed dense depth image. With the sparse depth image D t In D t The difference at the effective point is expressed as follows: Among them, L sd represents the sparse depth error, Ψ represents the sparse depth image D t The effective point set of the smoothing error is to use the image gradient to adjust the error weight of the depth gradient, and the expression is shown in the following formula (14): Among them, L sm represents the smoothing error; the total loss function L(θ) combines the photometric error, sparse depth error and smoothing error, and its expression is shown in the following formula (15): L(Θ)=L p h+λ sd L sd +λ sm L sm (15) Among them, Θ represents the parameters of the repair network model, λ sd and λ sm are the regularization parameters for the sparse depth error and smooth error, respectively.

Images