A deep image inpainting method and system based on attention-guided gating network

Abstract

The application provides a kind of depth image completion method and system based on attention guide gate network.The method comprises: obtaining RGB color map, obtaining original depth map corresponding to RGB color map;The RGB color map and the original depth map are input into the first depth map completion network, and the output of the first depth map completion network is obtained as the rough filled depth image;The original RGB color map is input into the standard four convolution layer to extract each layer color feature;The rough filled depth image is input into the second depth completion network, and the output of the second depth completion network is obtained as the completed depth image.The application uses RGB color map to complete the missing depth map based on a kind of attention guide gate network, so as to obtain more complete and more accurate depth image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular, especially relates to a depth image completion method and system based on attention guided gating network. BACKGROUND

[0002] Depth perception is crucial in applications such as autonomous driving, robot navigation, and scene reconstruction. Commonly used depth sensors include lidar, time-of-flight sensors, or stereo cameras. However, most obtained depth images inevitably come with many invalid regions caused by reflections, too far distances, light effects, or other environmental noise. These invalid data will have a serious adverse effect on the application process of depth information. Therefore, depth completion is necessary for most depth image-based applications.

[0003] Currently, there are mainly two types of completion algorithms for depth missing in the field of depth completion. The first type is the traditional method, that is, to design the corresponding operation method according to the typical characteristics of depth in different scenes, such as joint bilateral filtering, fast marching, and Markov random field method. The characteristic of this method is stable results, but the calculation process is complex, the operation speed is slow, and the real-time performance is poor. The second type is the depth completion algorithm based on deep learning. This method generally uses a specially designed convolutional neural network as the main model, and is trained with a calibrated and corrected data set. Its characteristic is that it has equivalent or better depth completion results than traditional methods in limited application scenarios, and relies on some hardware with high-performance parallel computing capabilities to achieve a much faster calculation speed than traditional methods. The method based on deep learning usually uses a convolutional autoencoder to extract depth and color features from RGB-D data, and fuses them to complete the depth map.

[0004] Although the existing method based on deep learning has good effect, it still has some problems in logic. First, the standard convolution operation treats all inputs as valid values, while the original depth map contains a large number of invalid values, which may damage the accuracy of the features and cause various visual artifacts in the reconstruction process, such as holes, contradictions, and edge blurring. In addition, most existing models use color information for guidance, that is, using a convolutional layer to extract color features, and then directly concatenating them with depth features to realize the fusion of color and depth information. However, the color features contain a lot of information that is irrelevant to depth, which may have a negative impact on the depth completion result in the depth reconstruction process. Therefore, most of the existing depth completion methods have optimization space for the above problems. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a depth image completion method and system based on an attention-guided gating network. First, the corresponding color and depth images are obtained and processed from the used data set, then a UNet convolutional auto-encoding network is used as a depth filling module for zero value filling, then a UNet-like double-branch convolutional auto-encoding network is used as a depth reconstruction module to obtain accurate and complete depth images, and finally an edge-preserving-based completion evaluation module is used to guide the auto-encoding network to obtain more refined completion results. The present application can effectively obtain more complete and accurate depth images based on the original depth image and the corresponding RGB image.

[0006] The technical means adopted by the present application are as follows:

[0007] In one aspect, the present application discloses a depth image completion method based on an attention-guided gating network, comprising the following steps:

[0008] S1, obtaining an RGB color image and an original depth image corresponding to the RGB color image;

[0009] S2, inputting the RGB color image and the original depth image into a first depth image completion network to obtain the output of the first depth image completion network as a roughly filled depth image, the first depth image completion network being used to fill the missing area of the original depth image through an encoder combined with the RGB color image;

[0010] S3, inputting the original RGB color image into a standard four convolutional layer to extract color features of each layer;

[0011] S4, inputting the roughly filled depth image into a second depth completion network to obtain the output of the second depth completion network as a completed depth image, the second depth image completion network being used to reconstruct the roughly filled depth image through an encoder combined with the color features, the second depth image completion network comprising four layers of depth encoding network and four layers of depth decoding network, and each layer of color feature being input as attention guidance of the corresponding depth encoding layer and depth decoding layer.

[0012] Further, the RGB color image and the original depth image are input into the first depth image completion network to obtain the output of the first depth image completion network as a roughly filled depth image, comprising:

[0013] The RGB color image and the original depth image are spliced into a four-channel multi-modal feature image;

[0014] The four-channel multi-modal feature image is input into two layers of standard convolutional layers to extract features, and then input into two layers of standard deconvolutional layers to reconstruct the depth image;

[0015] The rough filled depth image is input into a second depth completion network to obtain an output of the second depth completion network as a completed depth image, including:

[0016] Further, the rough filled depth image is input into a second depth completion network to obtain an output of the second depth completion network as a completed depth image, including:

[0017] A color encoder is established, color features are extracted using a standard convolutional layer, and color feature images of different sizes are output layer by layer;

[0018] A depth encoder is established, more accurate depth features are extracted using an attention-guided gated convolution combined with color feature images of corresponding sizes, and depth feature images of different sizes are output layer by layer;

[0019] A depth decoder is established, the output of the deepest layer of the color encoder and the depth encoder is spliced as the input of the first layer of the decoder, a depth joint feature is reconstructed using a gated transposed convolution, the depth joint feature includes a part of the output of the previous layer, a depth skip connection part of corresponding size and a color skip connection part, and the color skip connection part uses an attention-guided skip connection.

[0020] Further, the depth image completion method further includes the following steps:

[0021] S5, the first depth map completion network and the second depth map completion network are jointly trained according to the difference between the complete depth image and the pre-prepared depth image.

[0022] Further, after obtaining the original RGB color image and the original depth image, the method further includes: pre-processing the obtained original RGB color image and the original depth image, and the pre-processing includes image alignment, data normalization and data augmentation.

[0023] In one aspect, the application discloses a depth image completion system based on an attention-guided gated network, including:

[0024] An acquisition unit is configured to acquire an RGB color image and an original depth image corresponding to the RGB color image;

[0025] A first completion unit is configured to input the RGB color image and the original depth image into a first depth map completion network to obtain an output of the first depth map completion network as a rough filled depth image, and the first depth map completion network is configured to fill a missing area of the original depth image through an encoder combined with the RGB color image.

[0026] A color feature extraction unit is configured to input the original RGB color image into a standard four-convolutional layer to extract color features of each layer;

[0027] The second completion unit is configured to input the roughly filled depth image into a second depth completion network to obtain an output of the second depth completion network as a completed depth image, the second depth completion network is configured to reconstruct the roughly filled depth image by an encoder joint color feature, and the second depth completion network comprises four layers of depth encoding networks and four layers of depth decoding networks, and each layer of color feature is taken as an attention guide input of a corresponding depth encoding layer and a depth decoding layer.

[0028] In one aspect, the application discloses a storage medium, which comprises a stored program, wherein the program, when executed, performs the depth image completion method based on the attention guide gating network according to any one of the above.

[0029] In another aspect, the application discloses an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor performs the depth image completion method based on the attention guide gating network according to any one of the above by executing the computer program.

[0030] Compared with the prior art, the application has the following advantages:

[0031] In order to solve the defects of most depth completion methods, the application first acquires and processes corresponding color and depth images from a used data set, then uses a UNet convolutional auto-encoding network as a depth filling module for zero value filling, and then uses a UNet-like double-branch convolutional auto-encoding network to obtain an accurate and complete depth image, the UNet-like double-branch convolutional auto-encoding network comprises a depth feature encoder and a depth feature decoder. The above feature encoder comprises an attention guide gating convolution AG-GConv to realize cross-modal feature fusion of different scales. The above feature decoder comprises a gating deconvolution De-GConv and an attention guide skip connection AG-SC to avoid introducing too many depth-independent features in the reconstruction process, and finally based on the edge preservation result to guide the auto-encoding network to obtain a more fine completion result. The two neural networks of the application are jointly trained, and then a more complete and accurate depth image is effectively obtained based on the original depth image and the corresponding RGB image. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0033] Figure 1A flow chart of a depth image completion method based on an attention guided gating network according to the present application.

[0034] Figure 2 A network architecture diagram used by a depth image completion method based on an attention guided gating network according to the present application.

[0035] Figure 3 An attention guided gating convolution AG-GConv structure diagram according to the present application, (a) represents an attention guided gating convolution AG-GConv, and (b) represents an attention guide CA.

[0036] Figure 4 An attention guided skip connection AG-SC structure diagram according to the present application, (a) represents an attention guided skip connection AG-SC, and (b) represents a gating deconvolution De-GConv.

[0037] Figure 5 A schematic diagram of four groups of depth completion results in an embodiment of the present application.

[0038] Figure 6 A system structure block diagram of a depth image completion system based on an attention guided gating network according to the present application. DETAILED DESCRIPTION

[0039] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0040] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] As Figures 1-5 shown, the present application provides a depth image completion method based on an attention guided gating network, comprising the following steps:

[0042] S1, acquire an RGB color image, and acquire an original depth image corresponding to the RGB color image.

[0043] The RGB color image acquired in the present application can be an RGB color image taken in a current scene, or an RGB color image acquired from a historical data set. Correspondingly, the original depth image can be a depth image of the current scene collected by a depth sensor, or a preprocessed depth image in the data set.

[0044] Before the RGB color image and the original depth image are completed, batch processing of the above data is further included, including image alignment, data normalization, data augmentation, etc. Specifically, it includes:

[0045] S101, data processing of the original depth image: in the training process, first, the image is randomly rotated within a range of-5 to 5 degrees, randomly centered cropped within a range of 50% to 80%, and adjusted to a fixed shape; then, a certain area of the image is randomly erased and replaced with empty pixels; finally, all pixel values of the image are mapped to the interval [0, 1]. In the testing process, the image is centered cropped to 50%, then adjusted to a fixed shape, and mapped to the interval [0, 1].

[0046] S102, data processing of the depth ground truth image: in the training process, first, the image is rotated using the same parameters as the above process, centered cropped using the same parameters as the above process, and adjusted to a fixed shape; then, all pixel values of the image are mapped to the interval [0, 1]. In the testing process, the image is centered cropped to 50%, then adjusted to a fixed shape, and mapped to the interval [0, 1].

[0047] S103, data processing of the color image: in the training process, first, the image is rotated using the same parameters as the above process, centered cropped using the same parameters as the above process, and adjusted to a fixed shape; then, the brightness, contrast and saturation of the image are randomly changed to simulate different shooting conditions and environments; finally, all pixel values of the image are standardized. In the testing process, the image is centered cropped to 50%, then adjusted to a fixed shape, and all pixel values of the image are standardized.

[0048] S2, input the RGB color image and the original depth image into a first depth image completion network to acquire an output of the first depth image completion network as a roughly filled depth image, the first depth image completion network being used to fill a missing area of the original depth image through an encoder combined with the RGB color image.

[0049] In the application, a UNet convolutional auto-encoding network is established, and the processed RGB and depth image with noise and holes are input to fill the missing area of the original depth image corresponding to the RGB picture while keeping the existing depth value unchanged, thereby providing a rough filling depth image of zero value pixels or depth value missing pixels. Specifically, the following steps are included:

[0050] S201, the original depth image and color image are spliced into a four-channel multi-modal feature image;

[0051] S202, the four-channel multi-modal feature image is sent to two standard convolution layers to extract features, and then sent to two standard deconvolution layers to reconstruct the depth image, and the convolution layer uses a larger size convolution kernel to ensure that the convolution receptive field is large enough to cover the largest invalid area;

[0052] S203, the original depth image and the reconstructed depth image are merged to fill only the missing area in the original depth image, keeping the existing depth value unchanged, thereby providing a rough filling depth image of zero value pixels or depth value missing pixels.

[0053] S3, the original RGB color image is input into a standard four convolution layer to extract color features of each layer.

[0054] In the application, standard convolution layers are used to extract color features, and different size color feature images are output layer by layer.

[0055] S4, the rough filling depth image is input into a second depth completion network to obtain the output of the second depth completion network as a completed depth image, the second depth image completion network is used to reconstruct the rough filling depth image through an encoder combined with color features, and the second depth image completion network includes four depth encoding networks and four depth decoding networks, and each layer of color features is input as attention guidance of the corresponding depth encoding layer and depth decoding layer.

[0056] In the application, a UNet-like double-branch convolutional auto-encoding network is used to reconstruct the depth image combined with the corresponding RGB picture, thereby fine-tuning the depth value of all zero value and depth value missing pixels to obtain a more complete and accurate depth image. The network includes three parts, in addition to the color feature encoder constructed by the standard convolution layer mentioned above, it also includes a depth feature encoder and a depth feature decoder, wherein:

[0057] The depth feature encoder is configured to extract more accurate depth features using attention guided gated convolution AG-GConv, which includes a standard convolution layer and an attention guided gating strategy that uses a context attention based on depth and color features to generate a pixel mask to gate the depth features, thereby improving the quality of the depth features.

[0058] The color feature encoder is configured to extract color features using a standard convolution layer.

[0059] The depth feature decoder is configured to reconstruct a depth joint feature using a gated de-convolution De-GConv, which includes an output part of the previous layer, a depth skip connection part, and a color skip connection part that uses an attention guided skip connection AG-SC, i.e., an attention modulated skip connection based on depth and color features, to suppress color features that are irrelevant to depth.

[0060] The specific depth map reconstruction steps include:

[0061] S401, a color encoder is established to extract color features using a standard convolution layer and output color feature images of different sizes layer by layer. The color encoder is composed of four groups of convolution feature extraction units with the number of output channels doubled step by step. Each unit includes two groups of convolution layers with step lengths of 1 and 2, two groups of BatchNorm regularization layers, and two groups of ReLU activation function layers. Each input color feature image is sequentially calculated in the order of convolution layer with step length 1, regularization layer, activation function layer, convolution layer with step length 2, regularization layer, and activation function layer, thereby obtaining the extracted color feature image

[0062] S402, a depth encoder is established to extract more accurate depth features using attention guided gated convolution AG-GConv on corresponding size color feature images, and output depth feature images of different sizes layer by layer.

[0063] The depth encoder is composed of four groups of attention guided gated convolution AG-GConv feature extraction units with the number of output channels doubled step by step. Specifically, as shown in Figure 3 each attention guided gated convolution AG-GConv unit includes two groups of convolution layers with step lengths of 1 and 2, two groups of BatchNorm regularization layers, and two groups of LeackyReLU activation function layers, an attention guide CA unit, and a gating unit. First, each input depth feature image The deep feature image is extracted by sequentially calculating a convolutional layer with a step size of 1, a regularization layer, an activation function layer, a convolutional layer with a step size of 2, a regularization layer, and an activation function layer in this order Then, the color feature image of the corresponding size is spliced along the channel axis direction Combined into joint features Next, the attention guide unit CA first calculates a new joint feature using a convolutional layer with a step size of 1, a regularization layer, and an activation function layer And the joint feature is split into C' slices along the channel axis direction Flattened into C' vectors L = H / 2 x W / 2, and then sequentially calculated in the order of a fully connected layer with an output vector length M = 4L, a ReLU activation function layer, a fully connected layer with an output vector length C', and Sigmoid, to obtain the reconstructed vector Finally, the vector g i Will be rearranged to obtain the gating feature Multiplied by the deep feature F d′ , thereby obtaining the extracted deep feature image

[0064] The purpose of the attention guide gated convolution is that for the original depth image with large holes, the pixels in the hole region are invalid, and the standard convolution cannot distinguish these invalid pixels from normal pixels, and they will be calculated in the same way on the entire image. Therefore, these extracted features are not reliable, and further, this unreliable feature will spread layer by layer, eventually seriously affecting the depth image reconstruction result. Although there are some special convolutions that can use gating signals to limit the spread of unreliable features, each convolution kernel can only consider the neighborhood of one pixel and cannot consider the background knowledge of the entire image. In contrast, the attention guide gated convolution AG-GConv considers both depth and color features and produces joint contextual attention on the spatial and channel axes, so the gating signal has higher reliability, and the unreliable feature is more effectively suppressed, which helps to improve the quality and reliability of the depth image reconstruction result.

[0065] S403, splice the outputs of the deepest layers of the two types of encoders as the input of the first layer of the decoder.

[0066] S404, establish a depth decoder, use a gated deconvolution De-GConv to reconstruct a depth joint feature, the depth joint feature includes a part of the output of the last layer, a depth skip connection part of the corresponding size, and a color skip connection part, the color skip connection part uses an attention guide skip connection AG-SC.

[0067] The depth decoder consists of four gated deconvolutions (De-Gconv) with the number of output channels halved at each stage. Specifically, as shown... Figure 4 As shown, each gated deconvolutional De-GConv unit consists of an upsampling layer using bilinear interpolation, a feature branch, and a gated branch. The feature branch consists of two sets of convolutional layers with a stride of 1, two sets of BatchNorm regularization layers, two sets of LeachyReLU activation function layers, and a gated unit; the gated branch consists of a set of convolutional layers with a stride of 1, a BatchNorm regularization layer, and a Sigmoid activation function layer. First, the joint feature image of each input... Including the output of the previous layer Depth jump connection part of corresponding size Connecting parts with color jumps The processed joint features are obtained through the upsampling layer. Then, the feature branches are calculated sequentially in the order of two sets of convolutional layers with a stride of 1, regularization layers, and activation function layers to obtain the feature image. The gated branch is calculated sequentially in the order of convolutional layer with stride of 1, regularization layer, and activation function layer to obtain the gated signal. Finally, the feature image F is multiplied by the gate signal G to obtain the reconstructed depth feature image.

[0068] During step S404, the color jump connection portion of the input joint features of the gated deconvolutional De-GConv unit is composed of an attention-guided jump connection AG-SC. Specifically, as shown... Figure 4 As shown, each attention-guided jump connection AG-SC consists of multiple sets of convolutional layers, batch normalization layers, and activation function layers. First, the deep features R output from the previous layer... d The feature image is obtained by sequentially calculating the convolutional layer with a kernel size of 1, the regularization layer, and the ReLU activation function layer. Color features F from the color encoder c The feature image is obtained by sequentially calculating the convolutional layer with a kernel size of 3, the regularization layer, and the ReLU activation function layer. Then the feature image R′ d and F′ c Merge according to the channel axis direction The gated signal is calculated sequentially in the following order: convolutional layer with kernel size 1, regularization layer, ReLU activation function layer, convolutional layer with kernel size 1, regularization layer, and Sigmoid activation function layer. Finally, color feature F c With gate signal G c Multiply to obtain attention-guided color jump connections.

[0069] The purpose of AG-SC is to consider the potential correlation between color and depth images in the same scene, and to use color features to assist in depth reconstruction, which is an effective method for depth completion. However, the correlation between color and depth is very complex and abstract. On the boundary of the object, the color pattern is usually closely related to the depth change; on the surface of the object, the depth change is usually stable, while the color and texture may change sharply. If the flat surface is a mirror with strong specular reflection or a blanket with low reflectivity, there may be a large area of holes in the corresponding area of the depth image. In these areas, color information may seriously mislead depth reconstruction. The AG-SC module aims to establish a local attention mechanism by learning the joint distribution of color and depth to suppress the color features irrelevant to depth in the skip connection and reduce their adverse effects on the reconstruction results of the depth image.

[0070] The purpose of De-Gconv is to merge the features of the encoder and the decoder to ensure the quality and accuracy of the decoding results. all d F can provide depth features of the previous layer of the encoder. d F'' can provide more detailed depth skip connection features. c F''' can provide color features filtered by AG-SC. However, their merging will still introduce some unreliable characteristics. De-GConv can filter out unreliable features by multiplying the original features with the gating signal to ensure the quality and accuracy of the decoding results.

[0071] S405, output the final depth completion image. It is composed of convolution layers with a step size of 1, BatchNorm regularization layers and Sigmoid activation function layers. Each input depth feature image is calculated in turn according to the order of convolution layers with a step size of 1, regularization layers and activation function layers, so as to obtain the final depth completion image.

[0072] Further, the depth image completion method further comprises the following steps:

[0073] S5, jointly train the first depth image completion network and the second depth image completion network according to the difference between the complete depth image and the pre-prepared depth image.

[0074] In the present application, an evaluation strategy for depth image completion results is established, which guides the training process of the depth completion network by comparing the difference between the completed depth image and the pre-prepared complete depth image. Specifically, it includes:

[0075] S501, calculate the image reconstruction loss according to the formula wherein,​ is the completed depth image, d i,j is the pre-completed depth image, M and N are the height and width of the reconstructed image respectively;

[0076] S502, calculate the edge preserving loss according to the formula where g v and g h are the vertical and horizontal gradients of the image respectively;

[0077] S503, calculate the overall image loss according to the formula where λ δ and λ p are the weights of the two types of image loss, typically λ δ = 0.7 and λ p = 0.3.

[0078] The purpose of establishing such an evaluation strategy is that the image reconstruction loss calculates the global consistency error of the image, the edge preserving loss calculates the boundary consistency error of the image, and the joint weighting strategy of the overall image loss balances the above two types of errors, so as to obtain a depth completion image with higher precision and higher quality.

[0079] The embodiment of the application uses the indoor data set NYU-Depth V2 as experimental data to verify the attention guided gating depth image completion method. NYU-Depth V2 is one of the most commonly used data sets for depth completion, containing 1449 combinations collected from 464 different indoor scenes, serving as a benchmark for evaluating the model and competing models of the application. The data set is randomly divided into 1029 groups of images for training and 420 groups of images for testing, and the original image size is adjusted to 324x288.

[0080] The model is implemented in Pytorch and trained using the SDG optimizer. In training and testing, the batch is set to 8, the initial learning rate is 0.01, and it is reduced to 30% when the loss tends to be stable, the momentum term is 0.95, and the weight decay term is 10 -4 , and the training is performed for 120 rounds.

[0081] For the evaluation indicators of the experiment, the model uses the following commonly used indicators: root mean square error RMSE, absolute relative error Rel and δ t , i.e. the percentage of predicted depth pixels within the threshold t = 1.10, 1.25, 1.25 2 and 1.25 3 , as a fine-grained evaluation.

[0082] Module effectiveness analysis: To analyze the effectiveness of each module, we conduct ablation experiments on the benchmark model and different schemes of the proposed modules using the NYU-Depth V2 dataset, as shown in Table 1. Scheme A is the basic framework of the model, which is a standard model similar to the network model containing standard convolution, standard deconvolution and skip connection. In scheme B, the fusion of depth and color is realized by directly splicing their feature images at each encoder layer. The results show that the addition of color features has obvious improvement on depth completion. Scheme C uses gated convolution GConv and gated deconvolution De-GConv instead of standard convolution and deconvolution, and performs better due to the filtering of unreliable features by the gating signal. Scheme D uses attention-guided gated convolution AG-GConv instead of gated convolution Gconv, and all evaluation indicators are significantly improved due to the introduction of context attention. Scheme E uses attention-guided skip connection AG-SC instead of color skip connection, and also restores gated convolution GConv and gated deconvolution De-GConv, and the improved performance proves the contribution of attention-guided skip connection AG-SC. Finally, in scheme F, the attention-guided gated convolution AG-Gconv and the attention-guided skip connection AG-SC are integrated together, and all three indicators reach the highest score, which is better than using any one module alone. All the above results show that the proposed attention-guided gated convolution AG-GConv and the attention-guided skip connection AG-SC are helpful to the depth completion task. Compared with gated convolution GConv, the advantage of attention-guided gated convolution AG-GConv lies in that the latter can learn the context attention of the gating signal based on depth and color features, while the former only uses depth features with local attention. The performance optimization of the proposed attention-guided skip connection AG-SC shows that filtering color features using local attention helps to reconstruct the depth image. In addition, their combination can further improve the final performance of the model. Compared with the benchmark model, the accuracy of the optimal scheme is significantly improved, with a decrease of 32.4% and 46.2% in RMSE and Rel, respectively, thus proving the effectiveness of each module.

[0083] Table 1 Ablation experiment results:

[0084]

[0085] Loss function effectiveness analysis: To analyze the effectiveness of each loss, we conduct different loss function ablation experiments using the NYU-Depth V2 dataset based on the optimal model, as shown in Table 2. Compared with the application of MSE loss, the application of Huber loss slightly improves all indicators, and the addition of edge preservation loss significantly improves the performance, thus proving the effectiveness of each loss.

[0086] Table 2 different loss function ablation experiment results:

[0087]

[0088] In another aspect, the present application discloses a depth image completion system based on attention guided gating network, as shown in the accompanying drawings, comprising: Figure 6

[0089] An acquisition unit is configured to acquire an RGB color image and an original depth image corresponding to the RGB color image.

[0090] A first completion unit is configured to input the RGB color image and the original depth image into a first depth image completion network to obtain an output of the first depth image completion network as a roughly filled depth image, and the first depth image completion network is configured to fill a missing area of the original depth image by an encoder jointing the RGB color image.

[0091] A color feature extraction unit is configured to input the original RGB color image into a standard four convolution layer to extract color features of each layer.

[0092] A second completion unit is configured to input the roughly filled depth image into a second depth completion network to obtain an output of the second depth completion network as a completed depth image, and the second depth completion network is configured to reconstruct the roughly filled depth image by an encoder jointing the color features, and the second depth completion network comprises four layers of depth encoding network and four layers of depth decoding network, and each layer of color features is taken as an attention guided input of a corresponding depth encoding layer and a depth decoding layer.

[0093] In another aspect, the present application discloses a storage medium comprising a stored program, wherein the program, when executed, performs the depth image completion method based on the attention guided gating network according to any one of the above aspects.

[0094] In another aspect, the present application discloses an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor performs the depth image completion method based on the attention guided gating network according to any one of the above aspects by executing the computer program.

[0095] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0096] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0097] ​In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other manners. Among them, the above-described device embodiments are only illustrative, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0098] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0099] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0100] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0101] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A depth image completion method based on attention-guided gating networks, characterized in that, Includes the following steps: S1. Obtain the RGB color image and the corresponding original depth image; S2. Input the RGB color image and the original depth image into the first depth image completion network, and obtain the output of the first depth image completion network as a coarsely filled depth image. The first depth image completion network is used to fill the missing areas of the original depth image by combining the encoder with the RGB color image. The RGB color image and the original depth image are input into a first depth image completion network, and the output of the first depth image completion network is obtained as a coarsely filled depth image, including: The RGB color image and the original depth image are stitched together to form a four-channel multimodal feature image. The four-channel multimodal feature image is input into two standard convolutional layers to extract features, and then input into two standard deconvolutional layers to reconstruct the depth image. The reconstructed depth image is used to fill in the missing regions in the original depth image, thus roughly filling the depth image. S3. Input the RGB color image into a standard four-convolutional layer to extract the color features of each layer; S4. Input the coarsely filled depth image into the second depth map completion network to obtain the output of the second depth map completion network as the completed depth image. The second depth map completion network is used to reconstruct the coarsely filled depth image by combining color features with the encoder. The second depth map completion network includes a four-layer depth encoding network and a four-layer depth decoding network, with the color features of each layer used as attention guidance inputs for the corresponding depth encoding and depth decoding layers, respectively. Inputting the coarsely filled depth image into the second depth map completion network to obtain the output of the second depth map completion network as the completed depth image includes: A color encoder is built, using standard convolutional layers to extract color features, and color feature images of different sizes are output layer by layer. A depth encoder is built, and attention-guided gated convolutions are used to extract more accurate depth features from color feature images of corresponding sizes. Depth feature images of different sizes are then output layer by layer. A depth decoder is established, and the deepest outputs of the color encoder and the depth encoder are concatenated as the first layer input of the decoder. Gated deconvolution is used to reconstruct the joint depth features, which include the output part of the previous layer, the depth jump connection part and the color jump connection part of the corresponding size, and the color jump connection part uses an attention-guided jump connection.

2. The depth image completion method based on attention-guided gating network according to claim 1, characterized in that, The depth image completion method further includes the following steps: S5. Jointly train the first depth map completion network and the second depth map completion network based on the difference between the complete depth image and the pre-made depth image.

3. The depth image completion method based on attention-guided gating network according to claim 1, characterized in that, After obtaining the original RGB color image and the original depth image, the process further includes: preprocessing the obtained original RGB color image and the original depth image, the preprocessing including image alignment, data normalization, and data augmentation.

4. A depth image completion system based on an attention-guided gating network, used to perform the depth image completion method based on an attention-guided gating network as described in any one of claims 1-3, characterized in that, include: The acquisition unit is used to acquire the RGB color image and the original depth image corresponding to the RGB color image; The first completion unit is used to input the RGB color image and the original depth image into the first depth image completion network, and obtain the output of the first depth image completion network as a coarsely filled depth image. The first depth image completion network is used to fill the missing areas of the original depth image by combining the encoder with the RGB color image. The RGB color image and the original depth image are input into a first depth image completion network, and the output of the first depth image completion network is obtained as a coarsely filled depth image, including: The RGB color image and the original depth image are stitched together to form a four-channel multimodal feature image. The four-channel multimodal feature image is input into two standard convolutional layers to extract features, and then input into two standard deconvolutional layers to reconstruct the depth image. The reconstructed depth image is used to fill in the missing regions in the original depth image, thus roughly filling the depth image. The color feature extraction unit is used to input the RGB color image into a standard four-convolutional layer to extract the color features of each layer. The second completion unit is used to input the coarsely filled depth image into a second depth map completion network, and obtain the output of the second depth map completion network as the completed depth image. The second depth map completion network is used to reconstruct the coarsely filled depth image by combining color features with an encoder. The second depth map completion network includes a four-layer depth encoding network and a four-layer depth decoding network, and uses the color features of each layer as the attention guidance input of the corresponding depth encoding layer and depth decoding layer, respectively. The process of inputting the coarsely filled depth image into the second depth map completion network and obtaining the output of the second depth map completion network as the completed depth image includes: A color encoder is built, using standard convolutional layers to extract color features, and color feature images of different sizes are output layer by layer. A depth encoder is built, and attention-guided gated convolutions are used to extract more accurate depth features from color feature images of corresponding sizes. Depth feature images of different sizes are then output layer by layer. A depth decoder is established, and the deepest outputs of the color encoder and the depth encoder are concatenated as the first layer input of the decoder. Gated deconvolution is used to reconstruct the joint depth features, which include the output part of the previous layer, the depth jump connection part and the color jump connection part of the corresponding size, and the color jump connection part uses an attention-guided jump connection.

5. A storage medium, characterized in that, The storage medium includes a stored program, wherein when the program is executed, it performs the depth image completion method based on an attention-guided gating network as described in any one of claims 1 to 3.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the depth image completion method based on attention-guided gating network as described in any one of claims 1 to 3 through the computer program.

Images