A depth map repair method and related apparatus based on generative adversarial networks

Abstract

According to embodiments of the present invention, a depth map inpainting method and related apparatus based on generative adversarial networks (GANs) are disclosed. Random noise is added to a real depth map to obtain a sparse depth map. The sparse depth map and its corresponding color image are input to a generator to generate a repaired depth map. The real or repaired depth map is then input to a discriminator to determine the probability that the input image is a real depth map. The determination result is backpropagated to the generator to update its network training parameters. Finally, the depth map to be repaired is input to the trained generator to generate the repaired depth map. Through the implementation of this invention, a GAN model consisting of a generator and a discriminator is trained based on a real depth map and a color depth map. Iterative training with updated parameters improves the discriminator's discrimination ability and the generator's inpainting ability, enabling the repair of depth maps with significant data loss and ensuring effective depth image inpainting.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a depth map inpainting method and related apparatus based on generative adversarial networks. Background Technology

[0002] In recent years, with the decreasing price of RGB-D image acquisition devices (such as Kinect, Realsense, and Xition) and their continuous improvement in accuracy and portability, depth information has been widely used in tasks such as image segmentation and object detection. While color images possess rich semantic information, depth maps offer rich texture information, providing effective assistance in various tasks. However, depth maps acquired by depth cameras like Realsense contain noise. Directly using these maps can negatively impact detection, segmentation, and planar mapping, making depth map restoration essential.

[0003] With the continuous maturation of deep learning, the research field of deep image inpainting has gradually achieved good results. Currently, there are many methods for deep image inpainting, including traditional filtering methods and deep learning-based methods. However, these methods can repair depth images with relatively few missing data, but cannot repair depth images with large missing data, resulting in poor image inpainting performance. Summary of the Invention

[0004] The main objective of this invention is to provide a depth image inpainting method and related apparatus based on generative adversarial networks, which can at least solve the problem of poor image inpainting effect of depth image inpainting algorithms provided in related technologies.

[0005] To achieve the above objectives, a first aspect of the present invention provides a depth map repair method based on generative adversarial networks, the method comprising:

[0006] Random noise is added to the real depth map to obtain a sparse depth map. The sparse depth map and the corresponding color map are then input into the generator to generate a repaired depth map.

[0007] The real depth map or the repaired depth map is input into the discriminator to determine the probability that the input image is the real depth map;

[0008] The discrimination result is backpropagated to the generator to update the network training parameters of the generator.

[0009] The depth map to be repaired is input into the generator that has been trained to generate the repaired depth map.

[0010] To achieve the above objectives, a second aspect of the present invention provides a depth map repair apparatus based on a generative adversarial network, the apparatus comprising:

[0011] The first generation module is used to add random noise to the real depth map to obtain a sparse depth map, and input the sparse depth map and the corresponding color map into the generator to generate a repaired depth map.

[0012] The discrimination module is used to input the real depth map or the repaired depth map into the discriminator and determine the probability that the input image is the real depth map;

[0013] The update module is used to backpropagate the discrimination result to the generator and update the network training parameters of the generator;

[0014] The second generation module is used to input the depth map to be repaired into the trained generator to generate the repaired depth map.

[0015] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method provided in the first aspect of this application.

[0016] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the method provided in the first aspect of this application.

[0017] According to the generative adversarial network-based depth map inpainting method and related apparatus provided in embodiments of the present invention, random noise is added to a real depth map to obtain a sparse depth map. The sparse depth map and its corresponding color image are input to a generator to generate a repaired depth map. The real depth map or the repaired depth map is input to a discriminator to determine the probability that the input image is a real depth map. The discrimination result is backpropagated to the generator to update the generator's network training parameters. The depth map to be repaired is input to the trained generator to generate a repaired depth map. Through the implementation of the present invention, a generative adversarial network model composed of a generator and a discriminator is trained based on a real depth map and a color depth map. By updating the parameters and iteratively training, the discrimination ability of the discriminator and the inpainting ability of the generator are improved, enabling the repair of depth maps with large data gaps and ensuring the depth image inpainting effect.

[0018] Other features and corresponding effects of the present invention will be described in the latter part of the specification, and it should be understood that at least some of the effects will become obvious from the description in the specification. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the basic process of the depth map repair method provided in the first embodiment of the present invention;

[0021] Figure 2 A schematic diagram illustrating the implementation principle of the coarse generator network provided in the first embodiment of the present invention;

[0022] Figure 3 A schematic diagram illustrating the implementation principle of feature fusion based on attention mechanism provided in the first embodiment of the present invention;

[0023] Figure 4 A schematic diagram illustrating the implementation principle of the fine generator network provided in the first embodiment of the present invention;

[0024] Figure 5 A schematic diagram illustrating the implementation principle of the multi-scale feature fusion module provided in the first embodiment of the present invention;

[0025] Figure 6 A schematic diagram illustrating the implementation principle of the joint attention module of channels and space provided in the first embodiment of the present invention;

[0026] Figure 7 A schematic diagram illustrating the implementation principle of the discriminator network provided in the first embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of the program modules of the depth map repair device provided in the second embodiment of the present invention;

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] First embodiment:

[0031] To address the technical problem of poor image restoration performance in existing depth image inpainting algorithms, this embodiment proposes a depth image inpainting method based on generative adversarial networks (GANs). This method is applied to a GAN model structure that includes a generator network and a discriminator network, such as... Figure 1 The diagram shown is a basic flowchart of the depth map repair method provided in this embodiment. The depth map repair method proposed in this embodiment includes the following steps:

[0032] Step 101: Add random noise to the real depth map to obtain a sparse depth map. Input the sparse depth map and the corresponding color map into the generator to generate the repaired depth map.

[0033] Specifically, in this embodiment, random noise is added to the dense depth map to obtain a sparse depth map, which is then used as input along with the corresponding color map. The generator then produces a dense depth map that is closest to the real one.

[0034] In one optional embodiment of this example, the generator includes a coarse generator network, which comprises a first encoding layer, a first intermediate layer, and a first decoding layer. The encoding layer includes a first sub-encoding layer and a second sub-encoding layer. Accordingly, the step of inputting the sparse depth map and the corresponding color image into the generator to generate the repaired depth map specifically includes: downsampling the sparse depth map through the first sub-encoding layer to extract first depth map features, and downsampling the color image through the second sub-encoding layer to extract color image features; weighting the features of different channels of the first depth map features and the color image features to obtain fused features; performing dilated convolution on the fused features through the first intermediate layer; and upsampling the dilated convolutioned fused features through the first decoding layer to generate the repaired depth map.

[0035] like Figure 2 The diagram illustrates the implementation principle of a coarse generator network provided in this embodiment. Specifically, the coarse generator network in this embodiment mainly repairs the global content of the depth map, including an encoding layer, an intermediate layer, and a decoding layer. It specifically involves 11 convolutional layers, 4 dilated convolutional layers, 2 deconvolutional layers, 2 stitching layers, and 6 fusion modules. The parameters of each module are shown in Table 1.

[0036] Table 1

[0037]

[0038]

[0039] In this embodiment, firstly, the depth map and color map after the superimposed noise mask are processed by the encoding layer part of convolutional layer 1-6 for feature extraction. Then, they pass through the intermediate layer composed of dilated convolution 1-4. Finally, they pass through the decoding layer composed of convolutional layer 7-11, deconvolutional layer 1-2 and splicing layer 1-2.

[0040] It should be understood that color images possess rich structural and textural information, and the color image information and depth image information in the same scene are related. Therefore, the synchronously acquired color image information can be used for depth image restoration. However, directly concatenating the depth image and color image for convolutional learning is unlikely to achieve satisfactory results. Therefore, this embodiment extracts features from both the depth image and the RGB image separately, and then fuses these two features using an attention-based fusion module to eliminate the impact of differences in the missing proportions of different depth images.

[0041] like Figure 3 The diagram illustrates the implementation principle of a feature fusion based on an attention mechanism provided in this embodiment. Feature fusion based on the channel attention mechanism involves weighting the features of different channels in the color image and depth image separately, and then summing the pixel values. The channel attention mechanism first uses a global average pooling step to compress a c-channel, h×w feature map into a c-channel, 1×1 feature map representing global information. Then, a fully connected layer reduces the number of channels to 1 / 16 of the original. After ReLU activation, another fully connected layer transforms the C / 16-dimensional vector back into a C-dimensional vector, and a Sigmoid activation (ensuring values ​​are between 0 and 1) is applied to obtain a weight matrix. Finally, this weight matrix is ​​multiplied by the original feature map to assign the weights, resulting in the fused feature.

[0042] Furthermore, this embodiment further explains the concatenation layer. In practical applications, high-level feature networks have a larger receptive field and stronger semantic information representation capabilities, but lower resolution and weaker geometric information representation capabilities; low-level feature networks have a smaller receptive field and higher resolution, but weaker semantic information representation capabilities and more noise. Therefore, during the first downsampling (convolutional layer 2), due to the shallow network layer, relatively low-level image features can be obtained. During the second downsampling (convolutional layer 4), as the network layer deepens, some more abstract and high-level image features have been extracted. In the decoder, corresponding upsampling occurs twice. In particular, the second upsampling is very close to generating the final image result. At this point, some low-level image features such as textures need to be generated. However, due to the deep network layer, the low-level image features extracted from the shallow layer may have been lost. Therefore, before deconvolution, the first downsampling layer in the encoder and the last upsampling layer in the decoder are concatenated, and the second downsampling layer is concatenated with the penultimate upsampling layer to supplement the low-level image features for the decoder. Then, the decoding operation continues through convolution.

[0043] Furthermore, in an optional embodiment of this example, the step of downsampling the sparse depth map through the first sub-coding layer to extract the first depth map features includes: performing two normal convolutions on the sparse depth map through the first sub-coding layer to obtain a normal convolution result feature map and a gated mask; and multiplying the normal convolution result feature map and the gated mask element-wise to obtain the first depth map features.

[0044] Specifically, learning both invalid and valid regions of the depth map simultaneously in ordinary convolution performs poorly, while gated convolution automatically generates masks for each layer through network learning, offering greater flexibility. The method involves performing two ordinary convolutions on the input depth map, yielding two outputs. One output is the normal convolution result feature map, while the other uses the same parameters but selects the sigmoid activation function to restrict the result to between 0 and 1, creating a gated mask. Finally, the feature map and the gated mask are multiplied element-wise to obtain the final depth map feature output. It should also be noted that in practical applications, since the input of a color image does not contain invalid regions, ordinary convolution is sufficient for the color image.

[0045] Furthermore, in another optional embodiment of this example, the generator further includes a fine generator network, which includes a second encoding layer, a second intermediate layer and a second decoding layer. The second encoding layer includes a third sub-encoding layer and a fourth sub-encoding layer. The second intermediate layer includes a global repair network, a local repair network and a joint attention module for channels and spaces. The local repair network includes a content-aware layer and a multi-scale feature fusion module.

[0046] Accordingly, after the above steps of upsampling the fused features after dilated convolution through the first decoding layer to generate the repaired depth map, the method further includes: downsampling the repaired depth map through the third sub-coding layer to extract the second depth map features, and downsampling the color image through the fourth sub-coding layer to extract the color image features; weighting the features of different channels of the second depth map features and the color image features to obtain fused features; performing dilated convolution on the fused features through a global repair network; performing content-aware calculation on the fused features through the content-aware layer of the local repair network to obtain the fuzzy repair result, and performing feature fusion on the fuzzy repair result through a multi-scale feature fusion module; inputting the dilated convolution result and the feature fusion result into a joint attention module of channels and space to obtain output features based on the channel and spatial attention mechanism; and upsampling the output features based on the channel and spatial attention mechanism through the second decoding layer to generate the final repaired depth map.

[0047] like Figure 4 The diagram illustrates the implementation principle of a fine generator network provided in this embodiment. Specifically, the fine generator network in this embodiment mainly repairs texture details. Its input is a color image and a depth map generated by a coarse generator. Similar to the coarse generator, it passes through an encoder, intermediate layers, and a decoder, ultimately outputting a single-channel depth map. In this embodiment, the color image and depth map are respectively processed through the encoding layer of convolutional layers 1-6 for feature extraction. After feature extraction at each layer, feature fusion based on an attention mechanism is performed. Then, the network passes through intermediate layers and finally through a decoding layer consisting of convolutional layers 7-11, deconvolutional layers 1-2, and concatenation layers 1-2. The specific parameters of the fine generator network are shown in Table 2.

[0048] Table 1

[0049]

[0050] In this embodiment, the difference between the fine generator and the coarse generator lies in the intermediate layer; therefore, this embodiment describes the intermediate layer in detail. The intermediate layer of the fine generator is divided into two paths: one is a global inpainting network, which includes dilated convolutions 1-4, and the other is a local inpainting network, which includes a content-aware layer and a multi-scale fusion module. The outputs of these two paths are then connected, and meaningful features are further extracted through a joint attention module of channels and spaces.

[0051] The content-aware layer divides the image into two parts: foreground and background. The foreground refers to the area to be repaired, while the background refers to other complete areas. Multiple 3×3 blocks are extracted from the known background of the image and used as convolution kernels for the foreground. Then, cosine similarity is calculated to select several blocks that are similar to the foreground area. The softmax method is used to score these blocks, and the block with the highest score is selected. The content of the foreground area is deconvolved from this block, resulting in a fine-grained blur repair result.

[0052] like Figure 5 The diagram illustrates the implementation principle of a multi-scale feature fusion module provided in this embodiment. This module extracts information features at different scales to better obtain the structural information of the image. First, the feature map is convolved with four kernels of different dilation rates to obtain J1-J4, which are 1, 2, 4, and 8 respectively. Then, J1 and J2 are added together and convolved 3×3 to obtain J5. J5 and J3 are added together and convolved 3×3 to obtain J6. J6 and J4 are added together and convolved 3×3 to obtain J7. Then, J1, J5, J6, and J7 are concatenated and convolved 1×1 to obtain J8. Finally, J8 and J1 are added together to obtain the final feature output.

[0053] like Figure 6 The diagram illustrates the implementation principle of a joint channel and spatial attention module provided in this embodiment. This module mitigates the impact of redundant information and better simulates the long-term dependencies between different regions in an image. First, the original feature map undergoes a channel attention mechanism, followed by a spatial attention mechanism. The spatial attention mechanism involves performing three 1×1 gated convolutions on the input. The features output from the first 1×1 gated convolution are transformed and transposed to obtain F1. The features output from the second 1×1 gated convolution are transformed to obtain F2. Matrix multiplication is performed on F1 and F2, and the softmax function is used to obtain the spatial attention matrix H. Then, the features output from the third 1×1 gated convolution are multiplied with H and transformed in dimension. Finally, the features are weighted at the pixel level with the original feature map to obtain the output features based on the channel and spatial attention mechanisms.

[0054] Step 102: Input the real depth map or the repaired depth map into the discriminator and determine the probability that the input image is a real depth map.

[0055] In a preferred embodiment of this invention, the discriminator includes a global discriminator and a local discriminator, which have receptive fields of different sizes. Therefore, in practical applications, the true depth map (i.e., the original dense depth map) or the restored depth map is input to the global discriminator and the local discriminator respectively, and the probability of the input image being a true depth map is determined at two different scales.

[0056] Specifically, to make the texture details of the repaired depth map more realistic and reasonable, this embodiment uses two network discriminators: a global discriminator and a local discriminator. The global discriminator takes the entire real depth image or the entire generated depth image as input and uses it to determine the probability that the generated image is a real image. The local discriminator takes a local region of the real image or a local region of the generated image as input and uses it to determine the probability that the locally generated image is a locally real image.

[0057] like Figure 7 The diagram illustrates the implementation principle of a discriminator network provided in this embodiment. The discriminator network structure consists of five convolutional layers and one fully connected layer. The local discriminator and the global discriminator have the same network structure. The outputs of the two discriminators are then concatenated to form a 1536-dimensional vector, which is then processed by a fully connected layer to ensure the output value is within the range of 0 to 1. This vector is used to calculate the probability that the image is a real image. The specific parameters of the discriminator network are shown in Table 3.

[0058] Table 3

[0059] Layer name convolution kernel Step length aisle Convolutional layer 1 5×5 2 48 Convolutional layer 2 5×5 2 96 Convolutional layer 3 5×5 2 192 Convolutional layer 4 5×5 2 384 Convolutional layer 5 5×5 2 384 Fully connected layer 768

[0060] Step 103: Backpropagate the discrimination result to the generator and update the generator's network training parameters.

[0061] Specifically, in this embodiment, the discrimination result of the discriminator is backpropagated to the generator to continuously update the parameters, so that the generator can generate more realistic depth maps. During the iterative training process, the discriminator will also improve its discrimination ability, ultimately achieving the effect that the depth map generated by the generator is of good quality, while the discriminator cannot distinguish whether the depth map comes from the depth map generated by the generator or the original real depth map.

[0062] Step 104: Input the depth map to be repaired into the trained generator to generate the repaired depth map.

[0063] In generative adversarial networks (GANs), as the network deepens, important spatial details are easily lost while acquiring high-level semantic features in the encoder-decoder structure. This embodiment addresses this by using low-level features as detailed information to guide the generation of high-level semantic features through a concatenation layer. A multi-scale feature fusion module extracts information features at different scales to better capture the structural information of the image. Furthermore, a joint channel and spatial attention module mitigates the impact of redundant information and better simulates the long-term dependencies between different regions in the image. Through the combined efforts of these modules, it achieves excellent restoration results even when repairing depth maps with large missing areas.

[0064] According to the depth map inpainting method based on generative adversarial networks provided in this invention, random noise is added to the real depth map to obtain a sparse depth map. The sparse depth map and its corresponding color image are input into a generator to generate a repaired depth map. The real depth map or the repaired depth map is input into a discriminator to determine the probability that the input image is a real depth map. The discrimination result is backpropagated to the generator to update the generator's network training parameters. The depth map to be repaired is input into the trained generator to generate a repaired depth map. Through the implementation of this invention, a generative adversarial network model composed of a generator and a discriminator is trained based on the real depth map and the color depth map. By updating the parameters and iteratively training, the discrimination ability of the discriminator and the inpainting ability of the generator are improved, enabling the repair of depth maps with large data gaps and ensuring the depth image inpainting effect.

[0065] Second embodiment:

[0066] To address the technical problem of poor image restoration performance in depth image inpainting algorithms provided in related technologies, this embodiment illustrates a depth map inpainting device based on generative adversarial networks (GANs), applied to the depth map inpainting method based on GANs described in the aforementioned embodiments. For details, please refer to [link to relevant documentation]. Figure 8 The depth map repair device in this embodiment includes:

[0067] The first generation module 801 is used to add random noise to the real depth map to obtain a sparse depth map, and input the sparse depth map and the corresponding color map into the generator to generate a repaired depth map.

[0068] The discrimination module 802 is used to input the real depth map or the repaired depth map into the discriminator and determine the probability that the input image is a real depth map;

[0069] Update module 803 is used to backpropagate the discrimination result to the generator and update the network training parameters of the generator;

[0070] The second generation module 804 is used to input the depth map to be repaired into the trained generator to generate the repaired depth map.

[0071] In one optional embodiment of this example, the generator includes a coarse generator network, which includes a first coding layer, a first intermediate layer, and a first decoding layer. The coding layer includes a first sub-coding layer and a second sub-coding layer.

[0072] Accordingly, when the first generation module performs the function of inputting the sparse depth map and the corresponding color map into the generator to generate the repaired depth map, it is specifically used to: downsample the sparse depth map through the first sub-encoding layer to extract the first depth map features, and downsample the color map through the second sub-encoding layer to extract the color map features; perform feature weighting on the features of different channels of the first depth map features and the color map features to obtain fused features; perform dilated convolution on the fused features through the first intermediate layer; and upsample the dilated convolutioned fused features through the first decoding layer to generate the repaired depth map.

[0073] Furthermore, in an optional implementation of this embodiment, when the first generation module performs the function of downsampling the sparse depth map through the first sub-coding layer to extract the first depth map features, it is specifically used to: perform two ordinary convolutions on the sparse depth map through the first sub-coding layer to obtain a normal convolution result feature map and a gated mask; and multiply the normal convolution result feature map and the gated mask element by element to obtain the first depth map features.

[0074] Furthermore, in another optional embodiment of this example, the generator further includes a fine generator network, which includes a second encoding layer, a second intermediate layer, and a second decoding layer. The second encoding layer includes a third sub-encoding layer and a fourth sub-encoding layer. The second intermediate layer includes a global repair network, a local repair network, and a joint attention module for channels and spaces. The local repair network includes a content-aware layer and a multi-scale feature fusion module.

[0075] Accordingly, the first generation module is also used to: downsample the repaired depth map through the third sub-encoding layer to extract the second depth map features, and downsample the color map through the fourth sub-encoding layer to extract the color map features; weight the features of different channels of the second depth map features and the color map features to obtain fused features; perform dilated convolution on the fused features through the global repair network; perform content-aware calculation on the fused features through the content-aware layer of the local repair network to obtain the blurred repair result, and perform feature fusion on the blurred repair result through the multi-scale feature fusion module; input the dilated convolution result and the feature fusion result into the joint attention module of channels and space to obtain the output features based on the channel and space attention mechanism; and upsample the output features based on the channel and space attention mechanism through the second decoding layer to generate the final repaired depth map.

[0076] In one optional implementation of this embodiment, the discriminator includes a global discriminator and a local discriminator, which have receptive fields of different sizes. Accordingly, the discrimination module is specifically used for the step of inputting a true depth map or a restored depth map into the discriminator and determining the probability that the input image is a true depth map, including: inputting the true depth map or the restored depth map into the global discriminator and the local discriminator respectively, and determining the probability that the input image is a true depth map at two different scales.

[0077] It should be noted that the depth map repair methods in the foregoing embodiments can all be implemented based on the depth map repair device provided in this embodiment. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the depth map repair device described in this embodiment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0078] The depth map inpainting device based on generative adversarial networks (GANs) provided in this embodiment adds random noise to the real depth map to obtain a sparse depth map. The sparse depth map and its corresponding color image are then input into a generator to generate a repaired depth map. The real or repaired depth map is then input into a discriminator to determine the probability that the input image is a real depth map. The determination result is backpropagated to the generator to update its network training parameters. Finally, the depth map to be repaired is input into the trained generator to generate the repaired depth map. Through the implementation of this invention, a GAN model consisting of a generator and a discriminator is trained based on the real depth map and the color depth map. Iterative training with updated parameters improves the discriminator's discrimination ability and the generator's inpainting ability, enabling the repair of depth maps with significant data loss and ensuring effective depth image restoration.

[0079] Third embodiment:

[0080] This embodiment provides an electronic device, see [link / reference] Figure 9 As shown, it includes a processor 901, a memory 902, and a communication bus 903, wherein: the communication bus 903 is used to realize the connection and communication between the processor 901 and the memory 902; the processor 901 is used to execute one or more computer programs stored in the memory 902 to implement at least one step in the depth map repair method in the first embodiment above.

[0081] This embodiment also provides a computer-readable storage medium, which includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, magnetic cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by a computer.

[0082] The computer-readable storage medium in this embodiment can be used to store one or more computer programs, which can be executed by a processor to implement at least one step of the method in Embodiment 1 above.

[0083] This embodiment also provides a computer program that can be distributed on a computer-readable medium and executed by a computing device to implement at least one step of the method in Embodiment 1 above; and in some cases, at least one step shown or described may be executed in a different order than that described in the above embodiments.

[0084] This embodiment also provides a computer program product, including a computer-readable device on which the computer program as shown above is stored. In this embodiment, the computer-readable device may include the computer-readable storage medium as shown above.

[0085] Therefore, those skilled in the art should understand that all or some of the steps, systems, and devices disclosed above, as well as the functional modules / units, can be implemented as software (which can be implemented using computer program code executable by a computing device), firmware, hardware, and suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as integrated circuits, such as application-specific integrated circuits (ASICs).

[0086] Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, computer program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. Therefore, this invention is not limited to any particular combination of hardware and software.

[0087] The above description, in conjunction with specific implementation methods, provides a further detailed explanation of the embodiments of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A depth map repair method based on generative adversarial networks, characterized in that, include: Random noise is added to the real depth map to obtain a sparse depth map. The sparse depth map and the corresponding color map are then input into the generator to generate a repaired depth map. The real depth map or the repaired depth map is input into the discriminator to determine the probability that the input image is the real depth map; The discrimination result is backpropagated to the generator to update the network training parameters of the generator. The depth map to be repaired is input into the generator that has been trained to generate the repaired depth map.

2. The depth map repair method as described in claim 1, characterized in that, The generator includes a coarse generator network, which includes a first coding layer, a first intermediate layer, and a first decoding layer. The coding layer includes a first sub-coding layer and a second sub-coding layer. The step of inputting the sparse depth map and the corresponding color map into the generator to generate the repaired depth map includes: The first depth map features are extracted by downsampling the sparse depth map through the first sub-coding layer, and the color map features are extracted by downsampling the color map through the second sub-coding layer. The features of different channels of the first depth map features and the color map features are weighted to obtain fused features; The fused features are subjected to dilated convolution through the first intermediate layer; The fused features after dilated convolution are upsampled by the first decoding layer to generate a repaired depth map.

3. The depth map repair method as described in claim 2, characterized in that, The step of downsampling the sparse depth map through the first sub-coding layer to extract the features of the first depth map includes: The sparse depth map is subjected to two ordinary convolutions through the first sub-encoding layer to obtain the normal convolution result feature map and the gated mask. The first depth map feature is obtained by multiplying the normal convolution result feature map and the gated mask element by element.

4. The depth map repair method as described in claim 2, characterized in that, The generator further includes a fine generator network, which includes a second encoding layer, a second intermediate layer and a second decoding layer. The second encoding layer includes a third sub-encoding layer and a fourth sub-encoding layer. The second intermediate layer includes a global repair network, a local repair network and a joint attention module for channels and spaces. The local repair network includes a content-aware layer and a multi-scale feature fusion module. After the step of upsampling the fused features after dilated convolution through the first decoding layer to generate a repaired depth map, the method further includes: The repaired depth map is downsampled and second depth map features are extracted through the third sub-coding layer, and the color map is downsampled and color map features are extracted through the fourth sub-coding layer. The features of different channels of the second depth map features and the color map features are weighted to obtain fused features; The fused features are subjected to dilated convolution through the global repair network; The fusion features are processed by the content-aware layer of the local repair network to obtain a fuzzy repair result, and the fuzzy repair result is fused by the multi-scale feature fusion module. The dilated convolution result and the feature fusion result are input into the joint attention module of the channel and space to obtain the output features based on the channel and space attention mechanism; The output features based on the channel and spatial attention mechanism are upsampled by the second decoding layer to generate the final repair depth map.

5. The depth map restoration method according to any one of claims 1 to 4, characterized in that, The discriminator includes a global discriminator and a local discriminator, and the global discriminator and the local discriminator have receptive fields of different sizes; The step of inputting the real depth map or the restored depth map into the discriminator and determining the probability that the input image is the real depth map includes: The true depth map or the restored depth map is respectively input to the global discriminator and the local discriminator to determine the probability that the input image is the true depth map at two different scales.

6. A depth map repair device based on generative adversarial networks, characterized in that, include: The first generation module is used to add random noise to the real depth map to obtain a sparse depth map, and input the sparse depth map and the corresponding color map into the generator to generate a repaired depth map. The discrimination module is used to input the real depth map or the repaired depth map into the discriminator and determine the probability that the input image is the real depth map; The update module is used to backpropagate the discrimination result to the generator and update the network training parameters of the generator; The second generation module is used to input the depth map to be repaired into the trained generator to generate the repaired depth map.

7. The depth map repair device as described in claim 6, characterized in that, The generator includes a coarse generator network, which includes a first coding layer, a first intermediate layer, and a first decoding layer. The coding layer includes a first sub-coding layer and a second sub-coding layer. When the first generation module performs the function of inputting the sparse depth map and the corresponding color map into the generator to generate the repaired depth map, it is specifically used for: The first depth map features are extracted by downsampling the sparse depth map through the first sub-coding layer, and the color map features are extracted by downsampling the color map through the second sub-coding layer. The features of different channels of the first depth map features and the color map features are weighted to obtain fused features; The fused features are subjected to dilated convolution through the first intermediate layer; The fused features after dilated convolution are upsampled by the first decoding layer to generate a repaired depth map.

8. The depth map repair device as described in claim 7, characterized in that, The generator further includes a fine generator network, which includes a second encoding layer, a second intermediate layer and a second decoding layer. The second encoding layer includes a third sub-encoding layer and a fourth sub-encoding layer. The second intermediate layer includes a global repair network, a local repair network and a joint attention module for channels and spaces. The local repair network includes a content-aware layer and a multi-scale feature fusion module. The first generation module is also used for: The repaired depth map is downsampled and second depth map features are extracted through the third sub-coding layer, and the color map is downsampled and color map features are extracted through the fourth sub-coding layer. The features of different channels of the second depth map features and the color map features are weighted to obtain fused features; The fused features are subjected to dilated convolution through the global repair network; The fusion features are processed by the content-aware layer of the local repair network to obtain a fuzzy repair result, and the fuzzy repair result is fused by the multi-scale feature fusion module. The dilated convolution result and the feature fusion result are input into the joint attention module of the channel and space to obtain the output features based on the channel and space attention mechanism; The output features based on the channel and spatial attention mechanism are upsampled by the second decoding layer to generate the final repair depth map.

9. An electronic device, characterized in that, include: Memory, processor, and communication bus; The communication bus is used to enable communication between the memory and the processor; The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images