Multi-scale fusion deep image enhancement method and device for RGB-D images
Through the multi-scale fusion network model combining the feature information of depth images and RGB images, the problems of depth noise, hollowness and discontinuity are solved, and high-quality depth image enhancement is achieved, which is significantly better than the existing technology.
Patent Information
- Application Number
- CN202110290784.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-18
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2041-03-18
AI Technical Summary
The prior art is difficult to effectively solve the problems of depth noise, depth cavity and depth discontinuity, resulting in poor depth image quality.
Using a multi-scale fusion network model, combining deep image processing branches and RGB image processing branches, the multi-scale fusion prediction branches gradually fuse feature information, predict enhanced depth images, and use mixed loss functions for network training to ensure high-quality image generation .
Significantly improve the quality of depth images, solve the problems of depth noise, depth voids and depth discontinuity, enhance the robustness and detail recovery capabilities of the image, and perform better than existing methods in terms of image accuracy and structural similarity. .
Smart Images
Figure CN113033645B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image processing technology, and in particular to a multi-scale fusion depth image enhancement method and device for RGB-D images. Background Art
[0002] Depth image enhancement technology can be divided into two categories. One is to improve the quality and accuracy of equipment or improve the design at the hardware level to obtain higher quality depth images. The other is to design algorithms based on image processing principles to enhance depth images from the software layer. Improving hardware requires considering the cost and physical factors, while improving software algorithms has low development costs and does not need to consider many restrictions, so the advantages are more obvious.
[0003] In recent years, deep learning has made remarkable progress in the field of traditional RGB image enhancement, and many ideas have been applied to the field of deep image enhancement. Jeon et al. chose the Laplacian pyramid deep network as the basic network structure and proposed LapDEN, which can generate clean and clear depth images from the original depth images, but cannot accurately restore large holes and object edges in the original images. Zhang et al. proposed using 3D reconstruction to generate a depth image dataset from an RGB-D stream. In contrast to the work of LapDEN, they mainly focused on estimating larger unobserved depth values, but were unable to eliminate noise and holes from low-quality RGB-D images. These methods are all dedicated to enhancing the quality of depth images captured by RGB-D cameras, but the key issues of depth noise, depth holes, and depth discontinuity have not been well resolved. Summary of the invention
[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a multi-scale fusion depth image enhancement method and device for RGB-D images that can solve the problems of depth noise, depth holes and depth discontinuity.
[0005] Technical solution: The multi-scale fusion deep image enhancement method of RGB-D images described in the present invention includes:
[0006] (1) establishing a multi-scale fusion network model, wherein the multi-scale fusion network model includes a depth image processing branch, an RGB image processing branch, and a multi-scale fusion prediction branch, wherein the depth image processing branch is used to extract feature information from the depth image, the RGB image processing branch is used to extract feature information from the RGB image paired with the depth image, and the multi-scale fusion prediction branch is used to gradually fuse the feature information extracted by the depth image processing branch and the RGB image processing branch in order from low scale to high scale, and predict an enhanced depth image;
[0007] (2) obtaining a number of image pairs including depth images and corresponding RGB images as samples, using the reference depth images after depth image enhancement as sample labels, inputting the established multi-scale fusion network model, and performing network training;
[0008] (3) The depth image to be enhanced and the corresponding RGB image are input into the trained multi-scale fusion network model to obtain the enhanced image.
[0009] Furthermore, the deep image processing branch in step (1) is specifically a residual learning network, including a first convolution module, a second residual module, a third residual module and a fourth residual module connected in sequence, and the scales of the four modules are reduced by 1 / 2 in sequence.
[0010] Furthermore, the RGB image processing branch in step (1) is specifically a fully convolutional network, which includes a first convolutional module, a second convolutional module, a third convolutional module and a fourth convolutional module connected in sequence, and the scales of the four modules are reduced by 1 / 2 in sequence. Each convolutional module contains three convolutional layers and a maximum pooling layer, the first convolutional layer is a hollow convolution, and the middle two convolutional layers are convolution kernels.
[0011] Furthermore, the multi-scale fusion prediction branch in step (1) includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module and a sixth convolution module connected in sequence, the scales of the first convolution module, the second convolution module, the third convolution module and the fourth convolution module are increased by a factor of 2 in sequence, and the scales of the fifth convolution module and the sixth convolution module are the same as the fourth convolution module.
[0012] Among them, the first convolution module includes two convolution layers, which respectively input the output of the fourth residual module in the depth image processing branch and the output of the fourth convolution module of the RGB image processing branch; the second convolution module includes two convolution layers and one transposed convolution layer, and the two convolution layers respectively input the output of the third residual module of the depth image processing branch and the output of the third convolution module of the RGB image processing branch; the third convolution module includes two convolution layers and one transposed convolution layer, and the two convolution layers respectively input the output of the second residual module of the depth image processing branch and the output of the second convolution module of the RGB image processing branch; the fourth convolution module includes two convolution layers and one transposed convolution layer, and the two convolution layers respectively input the output of the first convolution module of the depth image processing branch and the output of the first convolution module of the RGB image processing branch.
[0013] Furthermore, the loss function used in step (2) for network training is:
[0014]
[0015]
[0016]
[0017] Where L represents the total loss, Indicates data loss, represents the structure preservation loss, y represents the depth image in the sample, represents the sample label, l=1, 2, 3, 4 represent the image scales from large to small, specifically the scales of the first convolution module, the second residual module, the third residual module and the fourth residual module of the depth image processing branch, respectively. represents the predicted depth image at scale l, y l represents the real depth image of scale l, H represents the height of the input image, W represents the width of the input image, They represent y, The difference between the pixel in the i-th row and the j-th column and its neighboring pixels on the left and above is the gradient value.
[0018] The multi-scale fusion depth image enhancement device of RGB-D images described in the present invention includes a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, the above method is implemented.
[0019] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0020] 1. The dual-branch design takes advantage of the correlation between RGB images and depth images. The rich features and object contour boundary information obtained from RGB images have an auxiliary effect on the enhancement of depth images.
[0021] 2. The method of gradual fusion of multi-scale high-dimensional features can gradually fill in details from low-resolution to high-resolution images, making the enhancement process more robust and less prone to problems such as blurred boundaries and depth noise. It solves the problems of depth noise, depth holes and depth discontinuity. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is an architecture diagram of the multi-scale fusion network model provided by the present invention;
[0023] Figure 2 It is a prediction effect diagram of the present invention;
[0024] Figure 3 This is a comparison diagram of the images predicted by the present invention and various methods;
[0025] Figure 4 It is a comparison diagram of desktop details between the present invention and various methods;
[0026] Figure 5 This is a comparison chart of the effects of different fusion methods and different Loss selections on deep image restoration;
[0027] Figure 6 This is a comparison chart of the impact of different inputs on deep image restoration. DETAILED DESCRIPTION
[0028] This embodiment provides a multi-scale fusion deep image enhancement method for RGB-D images, comprising the following steps:
[0029] (1) Establish a multi-scale fusion network model, which includes a depth image processing branch, an RGB image processing branch, and a multi-scale fusion prediction branch, such as Figure 1 As shown, the depth image processing branch is used to extract feature information from the depth image, the RGB image processing branch is used to extract feature information from the RGB image paired with the depth image, and the multi-scale fusion prediction branch is used to gradually fuse the feature information extracted by the depth image processing branch and the RGB image processing branch in order from low scale to high scale, and predict an enhanced depth image.
[0030] The deep image processing branch is specifically a residual learning network, including a first convolution module, a second residual module, a third residual module and a fourth residual module connected in sequence, and the scales of the four modules are reduced by 1 / 2 in sequence. The input of this branch is the original depth image. The difference between the original depth image and the reference image is mainly some holes and noise, but most of the low-frequency information between the two is similar. The residual learning network is used here, so that the rich feature information in the image can be extracted through the deep network, and the input is superimposed on the output through the jump connection to ensure that the information of the original depth image is not lost. The feature learned by the residual network only needs to be the difference between the original image and the reference image, which can converge faster. The residual module used in the present invention is the residual module used in ResNetv2 proposed in the document [He K, Zhang X, Ren S, et al. Identity mappings in deep residu-al networks [C] / / European conference on computer vision. Springer, Cham, 2016: 630-645.].
[0031] The RGB image processing branch is specifically a full convolutional network, which includes a first convolutional module, a second convolutional module, a third convolutional module and a fourth convolutional module connected in sequence. The scales of the four modules are reduced by 1 / 2 in sequence. Each convolutional module contains three convolutional layers and a maximum pooling layer. The first convolutional layer is a hole convolution, and the middle two convolutional layers are convolution kernels. The input of this branch is an RGB image paired with a depth image. According to the assumptions of Lambertian reflection and spherical harmonic illumination model, an RGB image can be decomposed into two parts: a reflectance map and a shading map, which are defined as follows:
[0032] I(R,D,L)=S(N(D),L)×R
[0033] Where I is the RGB image, D is the depth map, N is the normal of the object surface, L is the lighting condition, S is the illumination map obtained from the normal map N and the lighting condition L, and R is the reflection map. From the above formula, it can be seen that the RGB image contains part of the information of the depth image, and the two are highly correlated. Although the mapping relationship is complex, the deep network can learn to fit this mapping to achieve the purpose of RGB image-assisted depth image enhancement. However, the main contribution of RGB images to depth image enhancement is still the overall structural information and some features such as object contours, and the features that can be obtained from RGB images are very rich, so the fully convolutional network structure is selected here.
[0034] The multi-scale fusion prediction branch includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module and a sixth convolution module connected in sequence, the scales of the first convolution module, the second convolution module, the third convolution module and the fourth convolution module are increased by a factor of 2 in sequence, the fifth convolution module and the sixth convolution module have the same scale as the fourth convolution module, wherein the first convolution module includes two convolution layers, respectively inputting the output of the fourth residual module in the depth image processing branch and the output of the fourth convolution module of the RGB image processing branch, the second convolution module includes two convolution layers and a conversion The first convolutional layer is a convolutional layer, and the two convolutional layers respectively input the output of the third residual module of the depth image processing branch and the output of the third convolutional module of the RGB image processing branch. The third convolutional module includes two convolutional layers and one transposed convolutional layer, and the two convolutional layers respectively input the output of the second residual module of the depth image processing branch and the output of the second convolutional module of the RGB image processing branch. The fourth convolutional module includes two convolutional layers and one transposed convolutional layer, and the two convolutional layers respectively input the output of the first convolutional module of the depth image processing branch and the output of the first convolutional module of the RGB image processing branch. This branch mainly fuses the multi-scale high-dimensional features generated by the depth image processing branch and the RGB image processing branch to predict a high-quality depth image. Due to certain perspective distortion in the scene image, the visual size of different objects in the picture is different. However, the receptive field size of the convolutional layer is fixed. For visual elements with a larger scale in the image, the receptive field can only cover part of its area, which is prone to blurred boundaries; while for visual elements with a smaller scale, the receptive field will include other visual elements, which is easy to cause the problem of unclear distinction of small objects. Therefore, this paper adopts a method of gradual fusion of multi-scale high-dimensional features, which can gradually fill in details from low resolution to high-resolution images. Because the information content of depth images is not as rich as color images, it is necessary to reduce the feature dimensions of restored depth images. The high-dimensional features generated by the depth image processing branch and the RGB image processing branch are first reduced to 16-dimensional features through two convolutional layers, so as to achieve the effect of removing complex and useless features and retaining important features. After that, the two are superimposed and passed through the device convolution layer to generate a depth image with a resolution doubled compared to the original feature map. Finally, predicted depth images of three scales can be obtained.
[0035] (2) Obtain several image pairs including depth images and corresponding RGB images as samples, use the reference depth images after depth image enhancement as sample labels, input them into the established multi-scale fusion network model, and perform network training.
[0036] Among them, the loss function used in network training is:
[0037]
[0038]
[0039]
[0040] Where L represents the total loss, Indicates data loss. represents the structure preservation loss, y represents the depth image in the sample, represents the sample label, l=1, 2, 3, 4 represent the image scales from large to small, specifically the scales of the first convolution module, the second residual module, the third residual module and the fourth residual module of the depth image processing branch, ω l represents the weight coefficient of different scales l, which is set to [0.2, 0.4, 0.7, 1.0] in this embodiment. represents the predicted depth image at scale l, y l represents the real depth image of scale l, H represents the height of the input image, W represents the width of the input image, They represent y, The difference between the pixel in the i-th row and the j-th column and its neighboring pixels on the left and above is the gradient value.
[0041] It is a combination of the L1 loss function and the L2 loss function. Since the L2 loss function has the characteristics of fast derivative solution speed and sensitivity to outliers, this function is used in many image restoration tasks. However, since some reference images in the data set used in the present invention still have problems with holes or noise, if only the L2 loss function is used, the model may pay too much attention to local holes and noise, thereby affecting the generalization of the model. Therefore, the present invention adopts the L2 loss function on small-scale images to ensure that the overall image will not have large deviations, and uses the L1 loss function on large-scale images to ensure that image details can be restored. In addition, the depth image that the present invention focuses on has obvious discontinuities at the edges between the foreground and background areas. Conventional loss functions such as the L1 loss function and the L2 loss function are difficult to retain this discontinuity. Therefore The function mainly focuses on the gradient preservation value between adjacent pixels. Since the reference image itself will lose details when it is reduced in scale through bilinear interpolation, it is only used for the image with the largest scale.
[0042] (3) The depth image to be enhanced and the corresponding RGB image are input into the trained multi-scale fusion network model to obtain the enhanced image.
[0043] This embodiment also provides a multi-scale fusion depth image enhancement device for RGB-D images, including a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, the above method is implemented.
[0044] The following is a simulation verification of the present invention.
[0045] (1) Training details and parameter settings
[0046] The training image dataset for this experiment comes from SUNRGBD, which includes 3389 image pairs collected by Xtion devices, 1159 image pairs collected by RealSense devices, 1449 image pairs collected by KinectV1, and 3784 image pairs collected by KinectV2, for a total of 9781 image pairs. We selected 800 images from the NYUV2 dataset included in the dataset as the test set, and the remaining 8981 images as the training set. Before training, since the resolution of RGB-D images collected by different devices is different, all images (RGB images, depth images) are first center-cropped to an image of size 400×560 and used as the input of the network.
[0047] The experiment uses the Pytorch platform to train the model, uses the Adam optimizer, sets the initial learning rate to 0.0001, and sets each decay to 0.5 before the 2nd, 4th, 7th, 10th, and 15th iterations. The experimental batch size is 20, and the model is iterated 100 times in total. The experiment uses a server equipped with NVIDIA Tesla V100 for training, with a video memory of 32GB.
[0048] (2) Comparison of experimental results
[0049] First, the prediction results of the present invention are compared with the reference image. Figure 2 As shown, it can be seen that the present invention fills holes well and protects depth discontinuity between objects.
[0050] Secondly, the present invention is compared with several public models published in recent years, and the evaluation indicators used are RMSE (root mean square error), MAE (mean absolute error) and SSIM (structural similarity). RMSE and MAE are used to directly measure the accuracy of depth prediction, while SSIM measures the structural similarity between the predicted image and the reference image from an overall perspective. The advanced methods compared are: FCRN: a fully convolutional residual neural network for depth image prediction based on color images; SharpNet: a multi-task prediction model based on color images, including depth prediction, normal estimation, and boundary contour prediction; LapDEN: a deep image restoration model based on a pyramid-like structure of a depth image; HFM-Net: a multi-level deep image prediction model based on color images and depth images, the original model is used for normal estimation, and the output is changed here to a depth image; SharpNet, which was originally proposed for multi-task generation of color images. But this is also its shortcoming. Using the same encoder and similar decoder to do three tasks cannot achieve targeted improvement of the problem of deep image restoration. The results of the present invention are also compared with the latest pyramid image restoration method LapDEN, which uses the image super-resolution concept of LapSRN to restore the depth image. The model of the present invention uses the fusion structure between color and depth images to enhance the depth image. In Table 1 (the best results are marked in bold black), it can be seen that the method of the present invention is superior to the existing methods in all indicators, whether in image accuracy or structural similarity. Figure 3 As shown in FIG. 1 , the method of the present invention is obviously closer to the reference image than the other two methods, and has great improvements in removing noise, filling holes, and retaining depth discontinuity. Figure 4 In the detailed display, it can be seen that due to the size and material of the objects on the desktop, it is relatively difficult for the device to collect data, and large areas of holes are prone to appear. Here, the LapDEN based on the depth image and the SharpNet prediction based on the RGB image have large deviations. The work of the present invention has better solved the problem of hole filling by integrating the features of RGB-D images, and also has a certain protection for the depth discontinuity of the boundary of the object. It can also be seen from Table 1 that the model of the present invention is also faster than other methods in terms of model running speed.
[0051] In addition, although the method of the present invention is supervised, it can still restore some image areas that are blurred in the reference image by fusing the features of the RGB image. Figure 2 From the faucet in the figure, it can be seen that the original depth image lacks depth values in the entire faucet part, and the reference image obviously smoothes the faucet part. However, the method of the present invention can clearly see the faucet part, and its depth value is close to the value of the entire faucet.
[0052] Table 1 Comparison of test set performance of various advanced methods
[0053]
[0054] (3) Ablation experiment
[0055] To investigate the impact of each network component and fusion scheme on the final performance, an ablation study is performed on the NYUV2 validation dataset. The quantitative comparisons are summarized in Table 2.
[0056] Table 2 Performance comparison of ablation experiments
[0057]
[0058] Selection of input branches: In order to determine whether the role of RGB images and depth images in the network is obvious, the two branches are deleted separately in the model, that is, the network becomes a simple encoding and decoding structure. From the comparison of the results of branches 'RGB', 'Depth' and 'RGB+Depth' in Table 2, it can be clearly seen that the network of 'RGB+Depth' is significantly better than the other two networks. Figure 6 It can also be seen that although the prediction effect of the single RGB branch is much worse than that of the single Depth branch, the model with the RGB branch has better repair capabilities for holes and noise areas. Because RGB images are highly correlated with depth images, they can play an auxiliary role in depth image enhancement.
[0059] Choice of feature fusion method: Existing methods usually use matrix addition (add) or concatenation (concatenate) for multimodal feature fusion. In order to compare them, by changing the feature fusion operation in the fusion branch, but keeping other network components and settings unchanged, the results are expressed as "addition" and "concatenation" respectively. It can be seen that the "addition" result is worse than the "concatenation" effect. This is because the features of RGB images and depth images are heterogeneous data from different sources, and addition means that the model will implicitly treat these two different data in the same way, which will lead to performance degradation. Concatenation means that the model does not destroy the independence of the original features, and can self-learn the weights of different features through new convolutional layers. Therefore, existing methods also use addition or concatenation operations according to the source and relevance of the features. From Figure 5 It can also be clearly seen from (gl) that the stitching and fusion method has a good effect on object recovery in depth images and protection of boundary discontinuity.
[0060] Choice of loss function: In addition to model changes, the choice of loss function will also have a certain impact on the final result. If the L1 loss function is used for all layers, the predicted image will be too much about the details, while using L2 will make the predicted image blurry. Figure 5 (df) It can be clearly seen that the use of the loss function (Hybrid loss) of the present invention not only retains the clarity of the overall structure, but also protects the depth discontinuity of the edge of the object.
[0061] From the above analysis, it can be seen that the multi-scale fusion network proposed in the present invention fuses the multi-scale RGB image features with the depth image features, which not only retains the structural information of the original depth image, but also combines the advantages of high resolution and rich features of RGB images, and solves the problems of holes and noise in the depth image. In addition, in view of the actual situation that the reference image of the data set is not perfect, the present invention also designs a hybrid loss function to ensure the generation of high-quality depth images. Extensive experimental results show that the method of the present invention is significantly superior to the existing advanced methods in the task of depth image enhancement. Ablation studies also prove that the RGB-D multi-scale fusion scheme proposed in the present invention is superior to the generation method of single RGB image or single depth image. The hybrid loss function in this paper also converges better than the L1 and L2 loss functions, and the results are better.
[0062] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A multi-scale fusion deep image enhancement method for RGB-D images, characterized in that The method includes: (1) establishing a multi-scale fusion network model, wherein the multi-scale fusion network model includes a depth image processing branch, an RGB image processing branch, and a multi-scale fusion prediction branch, wherein the depth image processing branch is used to extract feature information from the depth image, the RGB image processing branch is used to extract feature information from the RGB image paired with the depth image, and the multi-scale fusion prediction branch is used to gradually fuse the feature information extracted by the depth image processing branch and the RGB image processing branch in order from low scale to high scale, and predict an enhanced depth image; (2) obtaining a number of image pairs including depth images and corresponding RGB images as samples, using the reference depth images after depth image enhancement as sample labels, inputting the established multi-scale fusion network model, and performing network training; (3) Input the depth image to be enhanced and the corresponding RGB image into the trained multi-scale fusion network model to obtain the enhanced image; The deep image processing branch is specifically a residual learning network, including a first convolution module, a second residual module, a third residual module and a fourth residual module connected in sequence, and the scales of the four modules are reduced by 1 / 2 in sequence; The RGB image processing branch is specifically a full convolutional network, which includes a first convolutional module, a second convolutional module, a third convolutional module and a fourth convolutional module connected in sequence, the scales of the four modules are reduced by 1 / 2 in sequence, each convolutional module contains three convolutional layers and a maximum pooling layer, the first convolutional layer is a hole convolution, and the middle two convolutional layers are convolution kernels; The multi-scale fusion prediction branch includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module and a sixth convolution module connected in sequence. The scales of the first convolution module, the second convolution module, the third convolution module and the fourth convolution module increase by a factor of 2 in sequence. The scales of the fifth convolution module and the sixth convolution module are the same as the fourth convolution module. The first convolution module includes two convolution layers, which respectively input the output of the fourth residual module in the depth image processing branch and the output of the fourth convolution module of the RGB image processing branch. The second convolution module includes two convolution layers and a transposed convolution layer. The two convolution layers respectively input the output of the third residual module of the depth image processing branch and the output of the third convolution module of the RGB image processing branch. The transposed convolution layer The convolution layer is connected to the output of the first convolution module. The third convolution module includes two convolution layers and one transposed convolution layer. The two convolution layers respectively input the output of the second residual module of the depth image processing branch and the output of the second convolution module of the RGB image processing branch. The transposed convolution layer is connected to the output of the second convolution module. The fourth convolution module includes two convolution layers and one transposed convolution layer. The two convolution layers respectively input the output of the first convolution module of the depth image processing branch and the output of the first convolution module of the RGB image processing branch. The transposed convolution layer is connected to the output of the third convolution module. The fifth and sixth convolution modules are both one convolution layer. The fifth convolution module is connected to the output of the fourth convolution module, and the sixth convolution module is connected to the output of the fifth convolution module. The loss function used in step (2) for network training is: Where L represents the total loss, Indicates data loss, represents the structure preservation loss, y represents the real depth image in the sample, represents the depth image predicted by the model, Respectively represent the image scales from large to small, specifically the scales of the first convolution module, the second residual module, the third residual module and the fourth residual module of the depth image processing branch, ω l represents the weight coefficient of different scales l, represents the predicted depth image at scale l, y l represents the real depth image of scale l, H represents the height of the input image, W represents the width of the input image, They represent y, The difference between the pixel in the i-th row and the j-th column and its neighboring pixels on the left and above is the gradient value.
2. A multi-scale fusion deep image enhancement device for RGB-D images, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to claim 1 is implemented.