A collaborative restoration method for RGB-D images

Abstract

This invention proposes an RGB-D image co-inpainting method, relating to the technical field of image processing. It constructs an image dataset by combining RGB images, depth images, and edge images, then builds an RGB-D co-inpainting model. The model is trained using the image dataset to obtain a trained RGB-D co-inpainting model. Finally, the RGB image and depth image to be inpainted are input into the trained RGB-D co-inpainting model to obtain co-inpainted RGB and depth images. This method co-inpaints the RGB and depth images while utilizing edge images as auxiliary information, thus avoiding problems such as blurring, color distortion, and missing texture regions with artifacts in the output inpainted image, thereby improving the image restoration effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method for collaborative restoration of RGB-D images. Background Technology

[0002] Autonomous vehicles are one of the effective solutions for alleviating traffic congestion and minimizing traffic accidents. With the development of artificial intelligence, significant progress has been made in the field of autonomous vehicles. Generally, the software system of an autonomous vehicle includes the following modules: perception, localization, high-definition map (HD map), decision-making and planning, and control. Among these, the HD map module provides auxiliary information for the perception, localization, and navigation of autonomous vehicles and is a crucial component of the autonomous driving software system. During the creation of HD maps, the collected images and point clouds often contain dynamic objects, while high-definition maps require scenes without dynamic objects. Therefore, the process of removing dynamic objects and supplementing information in the corresponding areas, also known as map inpainting, is an indispensable technique in creating HD maps.

[0003] Currently, map restoration is mainly achieved through two methods: point cloud restoration and separate restoration of RGB and depth images. However, point cloud restoration requires significant computational power, while separate restoration of RGB and depth images can easily lead to image inconsistencies. Existing technology proposes a multi-scale fusion depth image enhancement method for RGB-D (RGB+Depth Map) images. This method employs a two-branch stepwise fusion scheme, allowing the RGB and depth image inputs to complement each other in depth prediction. Depth ensures the overall structural integrity of the image, while color fills in missing pixel values. Furthermore, a hybrid multi-scale loss function is designed by analyzing the noise distribution of real data, ensuring that high-quality, clear images can still be generated even when the real image data is noisy. However, while this scheme can achieve image enhancement, it cannot simultaneously restore RGB and depth images. Summary of the Invention

[0004] To address the current inability to simultaneously repair RGB and depth images, this invention proposes an RGB-D image collaborative repair method that utilizes edge images as auxiliary information to collaboratively repair RGB and depth images.

[0005] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows:

[0006] An RGB-D image collaborative restoration method includes:

[0007] S1. Collect RGB images, depth images, and corresponding edge images as an image dataset;

[0008] S2. Construct an RGB-D co-inpainting model, train the RGB-D co-inpainting model using an image dataset, and obtain a trained RGB-D co-inpainting model;

[0009] S3. Input the RGB image to be repaired, the depth image, and the corresponding edge image into the trained RGB-D co-inpainting model to obtain the co-inpainted RGB image and depth image.

[0010] This technical solution constructs an RGB-D collaborative restoration model, which simultaneously restores RGB images and depth images, and uses edge images as auxiliary information, effectively improving the image restoration effect.

[0011] Preferably, for the RGB images I∈R in the dataset 3×H×W Depth image D∈R H×W and edge image E∈R n×H×W The masking formula is as follows:

[0012] I M =I·M

[0013] D M =D·M

[0014] E M =E·M

[0015] Among them, I M D M E M Let M represent the RGB image, depth image, and edge image after the masking operation, respectively, where M ∈ {0, 1}. H×W .

[0016] Here, dynamic objects, such as people and cars, are removed from the image by performing a masking operation.

[0017] Preferably, the RGB-D collaborative repair model includes a generator and three discriminators;

[0018] The generator includes a feature extraction module, an FFC DenseLight module, and a pyramid module connected in sequence, used to repair RGB images, depth images, and edge images. The output of the pyramid module is connected to the input of the discriminator.

[0019] The discriminators are an RGB image discriminator, a depth image discriminator, and an edge image discriminator, which are used to score the input image and send back the parameters of the generator for adjustment.

[0020] Preferably, the feature extraction module includes a first branch for extracting RGB image features, a second branch for extracting depth image features, and a third branch for extracting edge image features;

[0021] The first branch and the second branch each include a normal convolution with a kernel size of 3, and the third branch includes a dilated convolution with a dilation factor of 2;

[0022] The extracted RGB image features, depth image features, and edge image features are added and fused together, and used as input to the FFCDenseLight module.

[0023] Here, based on I M D M E M To address modal diversity, different structural branches are designed in the feature extraction module for feature extraction.

[0024] Preferably, the FFC DenseLight module includes n sequentially connected FFC DenseNet blocks and n sequentially connected FFC DenseLightNet blocks. The FFC DenseLightNet block includes a local information extraction path and a global information extraction path. The extracted local information and global information are added and fused together, and then used as the input of the pyramid module.

[0025] Here, we leverage the global receptive field of FFC (Fast Fourier Convolution) in early layers to better extract features from images with large missing areas. Dynamic obstacles in road scene images may occupy a large number of pixels, and their removal may result in extensive masking. However, traditional convolutional networks do not have a sufficiently wide receptive field and are easily affected by local details or masking, leading to poor results. Therefore, we construct the FFCDenseLight module, which combines residual modules with FFC, to give the RGB-D co-inpainting model a larger receptive field in the initial layers.

[0026] Preferably, the pyramid module is a three-layer structure connected in sequence. The top layer outputs a small-sized RGB image and a small-sized depth image as auxiliary information for the middle layer; the middle layer outputs a medium-sized RGB image and a medium-sized depth image as auxiliary information for the bottom layer; and the bottom layer outputs a large-sized RGB image and a large-sized depth image as the collaboratively repaired RGB image and depth image.

[0027] Here, the pyramid module is used to emphasize structural information in an image by using a small image and to emphasize detailed textures by using a large image, thus finely restoring image details and better expressing image features.

[0028] Preferably, step S2 further includes constructing a perceptual loss L for the RGB image. per The steps include: inputting the RGB images from the image dataset and the repaired RGB images output by the generator into a pre-trained VGG network; summing the feature losses extracted from each layer of the VGG network to obtain the perceptual loss L of the RGB images. per L per The expression is as follows:

[0029]

[0030] in, Let N represent the expectation, and N represent the number of layers in the VGG network used for feature matching. This represents the repaired RGB image output by the generator. φ represents the true value of an RGB image. i This represents the image features extracted from the i-th layer of the VGG network.

[0031] Here, we construct the perceptual loss L of the RGB image. per This improves the restoration effect of RGB images.

[0032] Preferably, step S2 further includes constructing a feature matching loss L for the depth image. fm L fm The expression is as follows:

[0033]

[0034] Where N′ represents the number of layers in the depth image discriminator. This represents the repaired depth image output by the generator. σ represents the actual value of the depth image. j This represents the feature image extracted from the j-th layer of the depth image discriminator.

[0035] Here, we construct the feature matching loss L for depth images. fm This improves the restoration effect of depth images.

[0036] Preferably, step S2 further includes training the three discriminators, specifically including constructing the loss functions L for the three discriminators. adv，o L adv，o Convergence; L adv，o The calculation formula is as follows:

[0037]

[0038] Where DIS represents the discriminator, o∈(I,D,E).

[0039] Preferably, in step S2, the total loss function L of the RGB-D collaborative repair model is constructed. total The RGB-D co-integration model was trained until L total Convergence; L total The expression is as follows:

[0040] L total =λ 1，o L 1， o+λ per L per +λ fm L fm +λ adv，o L adv，o

[0041] Among them, L 1，o L represents the loss calculated by the discriminator between the generator's output image and the input image. per L represents the perceptual loss of an RGB image. fm L represents the feature matching loss of a depth image. adv，o λ represents the loss function of the discriminator. 1，o , λ per , λ fm and λ adv，o For hyperparameters;

[0042] L 1，o The calculation formula is as follows:

[0043]

[0044] Where λ1 and λ2 are hyperparameters.

[0045] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0046] This invention proposes an RGB-D image co-inpainting method. An image dataset is constructed using RGB images, depth images, and edge images. An RGB-D co-inpainting model is then built and trained using the image dataset. Finally, the RGB image and depth image to be inpainted are input into the trained RGB-D co-inpainting model to obtain the co-inpainted RGB and depth images. This method co-inpaints the RGB and depth images and utilizes the edge image as auxiliary information, avoiding problems such as blurring, color distortion, and missing texture regions with artifacts in the output inpainted image, thus improving the image restoration effect. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the RGB-D image collaborative restoration method proposed in Embodiment 1 of the present invention;

[0048] Figure 2 This diagram illustrates the structure of the RGB-D image collaborative restoration model proposed in Embodiment 2 of the present invention.

[0049] Figure 3 This is a schematic diagram of the FFC DenseLight module structure proposed in Embodiment 2 of the present invention. Detailed Implementation

[0050] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0051] To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions;

[0052] It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0053] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment proposes an RGB-D image collaborative restoration method, including:

[0057] S1. Collect RGB images, depth images, and corresponding edge images as an image dataset;

[0058] The edge image is obtained based on the RGB image and the depth image using the Canny edge detection algorithm.

[0059] For RGB images I∈R in the dataset 3×H×W Depth image D∈R H×W and edge image E∈R n×H×W The masking formula is as follows:

[0060] I M =I·M

[0061] D M =D·M

[0062] E M =E·M

[0063] Among them, I M D M E MLet M represent the RGB image, depth image, and edge image after the masking operation, respectively, where M ∈ {0, 1}. H×W .

[0064] S2. Construct an RGB-D co-inpainting model, train the RGB-D co-inpainting model using an image dataset, and obtain a trained RGB-D co-inpainting model;

[0065] The RGB-D collaborative repair model includes a generator and three discriminators;

[0066] The generator includes a feature extraction module, an FFC DenseLight module, and a pyramid module connected in sequence, used to repair RGB images, depth images, and edge images. The output of the pyramid module is connected to the input of the discriminator.

[0067] The discriminators are an RGB image discriminator, a depth image discriminator, and an edge image discriminator, which are used to score the input image and send back the parameters of the generator for adjustment.

[0068] S3. Input the RGB image to be repaired, the depth image, and the corresponding edge image into the trained RGB-D co-inpainting model to obtain the co-inpainted RGB image and depth image.

[0069] Example 2

[0070] Based on Example 1, such as Figure 2 As shown, the generator includes a feature extraction module, an FFCDenseLight module, and a pyramid module connected in sequence;

[0071] The feature extraction module includes a first branch for extracting RGB image features, a second branch for extracting depth image features, and a third branch for extracting edge image features;

[0072] The first branch and the second branch each include a normal convolution with a kernel size of 3, and the third branch includes a dilated convolution with a dilation factor of 2;

[0073] The extracted RGB image features, depth image features, and edge image features are added and fused together, and used as input to the FFCDenseLight module.

[0074] See Figure 3The FFC DenseLight module includes n sequentially connected FFC DenseNet blocks and n sequentially connected FFC DenseLightNet blocks. Each FFC DenseLightNet block includes a local information extraction path and a global information extraction path. The extracted local and global information are added and fused together, and then used as the input to the pyramid module. In this embodiment, the FFC DenseLight module includes 5 DenseNet blocks and 5 DenseLightNet blocks, where each block contains 4 FFC layers, with a growth rate of 12.

[0075] The pyramid module is a three-layer structure connected sequentially. The top layer outputs a small-sized RGB image and a small-sized depth image as auxiliary information for the middle layer. The middle layer outputs a medium-sized RGB image and a medium-sized depth image as auxiliary information for the bottom layer. The bottom layer outputs a large-sized RGB image and a large-sized depth image as the collaboratively repaired RGB image and depth image. Specifically, the dimensions of the RGB and depth images output by the top layer of the pyramid module are (1 / 4*H)*(1 / 4*W)*3 and (1 / 4*H)*(1 / 4*W), respectively; the dimensions of the RGB and depth images output by the middle layer are (1 / 2*H)*(1 / 2*W)*3 and (1 / 2*H)*(1 / 2*W), respectively; and the dimensions of the RGB and depth images output by the bottom layer are (H*W*3) and (H*W), respectively.

[0076] Example 3

[0077] Based on the above embodiments, during the training process, the total loss function L of the RGB-D collaborative repair model is constructed. total The RGB-D co-integration model was trained until L total Convergence; L total The expression is as follows:

[0078] L total =λ 1，o L 1，o +λ per L per +λ fm L fm +λ adv，o L adv，o

[0079] Among them, L 1，o L represents the loss calculated by the discriminator between the generator's output image and the input image. per L represents the perceptual loss of an RGB image. fm L represents the feature matching loss of a depth image. adv，o λ represents the loss function of the discriminator.1，o , λ per , λ fm and λ adv，o For hyperparameters;

[0080] Perceptual loss L for constructing RGB images per The steps include: inputting the RGB images from the image dataset and the repaired RGB images output by the generator into a pre-trained VGG network; summing the feature losses extracted from each layer of the VGG network to obtain the perceptual loss L of the RGB images. per L per The expression is as follows:

[0081]

[0082] in, Let N represent the expectation, and N represent the number of layers in the VGG network used for feature matching. This represents the repaired RGB image output by the generator. φ represents the true value of an RGB image. i This represents the image features extracted at the i-th layer of the VGG network; in this embodiment, the VGG network is a pre-trained VGG-19, i∈[1,19].

[0083] Constructing the feature matching loss L of the depth image fm L fm The expression is as follows:

[0084]

[0085] Where N′ represents the number of layers in the depth image discriminator. This represents the repaired depth image output by the generator. σ represents the actual value of the depth image. j This represents the feature image extracted from the j-th layer of the depth image discriminator.

[0086] During the training of the three discriminators, the loss functions L of the three discriminators are constructed. adv，o L adv，o Convergence; L adv，o The calculation formula is as follows:

[0087]

[0088] Where DIS represents the discriminator, and O∈(I,D,E).

[0089] The loss L calculated by the discriminator between the generator output image and the input image 1，o The calculation formula is as follows:

[0090]

[0091] Where λ1 and λ2 are hyperparameters.

[0092] Furthermore, in this embodiment, the ReLU function is selected as the activation function of the RGB-D co-integration model, and the Adam optimizer with a learning rate of 0.001 is used for optimization; the training period is set to 40, and the batch size is 16.

[0093] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for collaborative restoration of RGB-D images, characterized in that, include: S1. Collect RGB images, depth images, and corresponding edge images as an image dataset; S2. Construct an RGB-D co-inpainting model, train the RGB-D co-inpainting model using an image dataset, and obtain a trained RGB-D co-inpainting model; The RGB-D collaborative repair model includes a generator and three discriminators; The generator includes a feature extraction module, an FFC DenseLight module, and a pyramid module connected in sequence, used to repair RGB images, depth images, and edge images. The output of the pyramid module is connected to the input of the discriminator. The discriminators are an RGB image discriminator, a depth image discriminator, and an edge image discriminator, which are used to score the input image and send back the parameters of the generator for adjustment. The feature extraction module includes a first branch for extracting RGB image features, a second branch for extracting depth image features, and a third branch for extracting edge image features; The first branch and the second branch each include a normal convolution with a kernel size of 3, and the third branch includes a dilated convolution with a dilation factor of 2; The extracted RGB image features, depth image features, and edge image features are added and fused together, and used as the input to the FFCDenseLight module; The FFC DenseLight module includes n sequentially connected FFC DenseNet blocks and n sequentially connected FFCDenseLightNet blocks. The FFC DenseLightNet block includes a local information extraction path and a global information extraction path. The extracted local information and global information are added and fused together, and then used as the input of the pyramid module. S3. Input the RGB image to be repaired, the depth image, and the corresponding edge image into the trained RGB-D co-inpainting model to obtain the co-inpainted RGB image and depth image.

2. The RGB-D image collaborative restoration method according to claim 1, characterized in that, Also includes: RGB images I in the dataset Depth Image D and edge image E The masking formula is as follows: in, , , These represent the RGB image, depth image, and edge image after the masking operation, respectively, where M represents the mask. .

3. The RGB-D image collaborative restoration method according to claim 1, characterized in that, The pyramid module is a three-layer structure connected in sequence. The top layer outputs a small-sized RGB image and a small-sized depth image as auxiliary information for the middle layer. The middle layer outputs a medium-sized RGB image and a medium-sized depth image as auxiliary information for the bottom layer. The bottom layer outputs a large-sized RGB image and a large-sized depth image as the RGB image and depth image after collaborative repair.

4. The RGB-D image collaborative restoration method according to claim 3, characterized in that, Step S2 also includes constructing a perceptual loss for the RGB image. The steps include: inputting the RGB images from the image dataset and the repaired RGB images output by the generator into a pre-trained VGG network; summing the feature losses extracted from each layer of the VGG network to obtain the perceptual loss of the RGB images. , The expression is as follows: in, Expressing expectations, N i This indicates the number of layers used for feature alignment in the VGG network. This represents the repaired RGB image output by the generator. Represents the true value of an RGB image. This represents the image features extracted from the i-th layer of the VGG network.

5. The RGB-D image collaborative restoration method according to claim 4, characterized in that, Step S2 also includes constructing a feature matching loss for the depth image. , The expression is as follows: in, Indicates the number of layers in the depth image discriminator. This represents the repaired depth image output by the generator. Represents the actual value of the depth image. Indicates the depth image discriminator at the 1st j Feature images extracted from layers.

6. The RGB-D image collaborative restoration method according to claim 5, characterized in that, Step S2 also includes training the three discriminators, specifically constructing the loss functions for the three discriminators. ,make convergence; The calculation formula is as follows: Where DIS stands for discriminator. .

7. The RGB-D image collaborative restoration method according to claim 6, characterized in that, In step S2, the total loss function of the RGB-D co-integration model is constructed. Train the RGB-D co-integration model until... convergence; The expression is as follows: in, This represents the loss calculated by the discriminator between the generator's output image and the input image. This represents the perceptual loss of an RGB image. This represents the feature matching loss of a depth image. This represents the loss function of the discriminator; , , and For hyperparameters; The calculation formula is as follows: in, and This is a hyperparameter.

Images