A deep image super-resolution reconstruction method guided by dual feature fusion

Abstract

The present invention proposes a deep image super-resolution reconstruction method based on dual-feature fusion guidance, and the network model includes a feature extraction part and a depth recovery and reconstruction part. The feature extraction part uses the depth map after bicubic interpolation and the intensity map of the color map of the same scene as input, and uses the input pyramid to extract the depth features and intensity features step by step to obtain multi-scale features, and the obtained features are used as the input of the depth recovery and reconstruction part; the depth recovery and reconstruction part first uses the dual-channel fusion module to perform feature fusion on the extracted last-level depth features and intensity features, and then uses the dual-feature guided reconstruction module to use the depth features and intensity features obtained by the feature extraction part to guide the recovery and reconstruction of the previous level reconstruction features step by step, and finally obtains a depth map with good reconstruction effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to a deep image super-resolution reconstruction method based on a convolutional neural network. Background Art

[0002] Depth information is important information for humans to perceive things in three-dimensional space. The development of depth cameras can obtain the depth information of objects in space in real time, and is widely used in many fields such as robot vision, medical treatment, multimedia entertainment, etc. Due to the limitations of hardware and the influence of the imaging process, the depth images obtained by existing depth cameras have low resolution and poor quality, which in turn limits the development of many fields such as three-dimensional reconstruction and virtual reality. Therefore, the research on super-resolution reconstruction technology of depth images is of great significance. Among them, the super-resolution reconstruction technology of depth images is divided into two categories: super-resolution reconstruction of single depth maps and super-resolution reconstruction of depth images guided by color images.

[0003] The advantage of directly reconstructing a single depth map for super-resolution is that it requires less input and less computation, but it also means that less information can be used and there are serious pathologies. The color image of the same scene contains a large amount of high-frequency information (such as edge information of the image) to assist in the pixel prediction of the depth image. Therefore, the high-resolution color image of the same scene can be used as a data guide to assist in the reconstruction of the depth image. The development of convolutional neural network technology has also performed well in the current image super-resolution reconstruction, and its rich feature extraction is more conducive to image reconstruction. However, the existing convolutional neural network-based, color image-guided super-resolution reconstruction technology for deep images has problems such as insufficient feature fusion, texture transfer and depth loss in the reconstructed depth map that need to be solved. Summary of the invention

[0004] To address the above phenomenon, this paper provides a deep image super-resolution reconstruction network based on convolutional neural network.

[0005] To this end, the technical solution adopted by the present invention is:

[0006] A method for deep image super-resolution reconstruction network based on dual feature fusion guidance, training the entire network includes the following steps:

[0007] (1) Dataset preparation: A certain amount of depth images and color images of the same scene are selected from different public depth image datasets, and the obtained image data are enhanced and the input image is acquired.

[0008] (1.1) The depth image and the color image of the same scene are rotated and flipped by 90°, 180°, and 270°, and then overlapped and sampled with a step size of 48, and cropped to obtain image blocks of size 96×96. The obtained image blocks are used as the training set and validation set for network training.

[0009] (1.2) Downsample the enhanced depth images one by one to obtain a low-resolution depth map.

[0010] (2) Image preprocessing: Before network training, the low-resolution depth map and the high-resolution color map of the same scene need to be preprocessed, including: performing bicubic interpolation on the low-resolution depth map to expand the depth map to the same color map of the same scene. Figure 1 The same size is used for high-resolution color images of the same scene. Since the intensity information in the color image is used as a guide, only the Y channel is taken as the intensity image. These two are used as network inputs.

[0011] (3) Network structure design: The entire network structure is mainly divided into two parts.

[0012] The first part is the feature extraction part. Here, two identical input pyramid structures are used to extract the feature from the input depth map. Extract deep features from the intensity map Y hr Intensity features, and add residual attention modules to strengthen features in each level, and fuse them with the features of the next level to obtain multi-level features, and the features obtained in each layer will be used to guide recovery and reconstruction in the second part.

[0013] The second part is the depth recovery and reconstruction part. Here, the dual-channel fusion module is first used to perform dual-feature fusion on the last level of multi-level features obtained in the feature extraction part. The fused features are then processed sequentially through deconvolution, dual-feature guided reconstruction module, and residual attention module for depth recovery and reconstruction. The above cycle operation is repeated 4 times to obtain the final depth features, which are then pixel-added with the input depth map to obtain the final reconstructed depth map. Each level of features in the feature extraction part mentioned above will be used in the dual-feature guided reconstruction module in this part to guide the output features of the previous level for depth recovery and reconstruction.

[0014] In addition, the present invention uses the minimum mean error (MSE) as the loss function to shorten the gap between the depth map reconstructed by the network and the real depth map:

[0015]

[0016] Where k represents the kth image; N represents the number of training samples each time; θ represents the parameter to be learned; represents the input depth map, Represents the Y channel image of the corresponding color image; Represents the reconstructed depth map; Represents the corresponding real depth map.

[0017] (4) The entire network is trained using the data set obtained in step (1), and the network parameters are updated using the gradient descent method based on back-propagation.

[0018] (5) The test low-resolution depth image and the color image of the same scene are preprocessed and input into the trained network model to obtain a super-resolution depth map at the output layer.

[0019] Specifically, the feature extraction part in step (3):

[0020] The feature extraction part includes a depth feature extraction branch and an intensity feature extraction branch. The depth image and intensity image described in step (3) are used as inputs, and features of the two images are extracted using two consistent input pyramid structures. Take the depth feature extraction branch as an example.

[0021] For the input depth map Downsampling is performed. Each time the depth map after downsampling is convolved with different numbers of channels to obtain feature maps of multiple scales. After being concatenated with the features after the previous level of pooling, the enhanced multi-scale features are obtained through residual attention operation. The mathematical model of this process is:

[0022]

[0023]

[0024]

[0025]

[0026] Among them, i∈{1, 2, 3, 4}, is the depth image after the i-th downsampling; is from The features extracted from and are the weights and biases in the convolution operation; σ(·) represents the ReLU activation function; maxpool(·) represents the downsampling operation; f RAM (·) represents the residual attention operation; represent Features obtained after residual attention operation; cat(·) represents concatenation operation.

[0027] The deep recovery and reconstruction part in step (3) is:

[0028] The deep recovery and reconstruction part mainly includes a dual-channel fusion module and a dual-feature guided reconstruction module.

[0029] The deep features obtained at the last level of the feature extraction part and strength characteristics First, the fusion features are obtained through the dual-channel fusion module Then, using this as input, we first perform feature upsampling and channel compression through deconvolution, and the output result is the same level of deep features as the feature extraction part. and strength characteristics The two features are jointly entered into the dual-feature guided reconstruction module for guided recovery and reconstruction. The fused features are subjected to residual attention operation to obtain guided features. After three cycles of the above operation, the reconstructed features are obtained. The final features Do a 1×1 convolution and add it to the input depth map Add pixels to get the reconstructed depth map D sr The mathematical model of this process is:

[0030]

[0031]

[0032]

[0033] Where j∈{0, 1, 2, 3}, f DCM (·) represents the dual-channel fusion module; f DGM (·) represents the dual-feature guided reconstruction module; deconv(·) represents deconvolution; σ represents the ReLU activation function; W sr and b sr Represent the weight and bias of the 1×1 convolution operation respectively; Represents pixel addition.

[0034] Furthermore, in the dual-channel fusion module (DCM), the deep feature F obtained by the feature extraction part is d and intensity characteristic F g Perform channel attention operation, and then add them to the original features to achieve enhanced features and The two enhanced features are initially fused by concatenation and then convolution, and finally a channel attention operation is performed to complete the dual-channel feature fusion.

[0035] The mathematical model of this process is:

[0036]

[0037]

[0038]

[0039] Among them, f C (·) represents the channel attention operation; W DCM and b DCM are the weights and biases in the convolution operation.

[0040] Similarly, in the dual-feature guided reconstruction module (DGM), the depth features F d and intensity characteristic F g First, do self-selection connection SSC to obtain the feature F after dual feature fusion SSC , which is used to guide the previous level feature F s To restore and rebuild. The specific operation is to restore F SSC Features and F s The features are concatenated and convolved to obtain the initial fusion feature F mix Finally, perform another channel attention operation to fully integrate and obtain the output feature F out .

[0041] The mathematical model of this process is:

[0042] F SSC =f SSC (F d , F g ) (12)

[0043] F mix =σ(W DGM ·cat(F SSC , F s )+b DGM ) (13)

[0044] F out =f C (F mix ) (14)

[0045] Among them, f SSC (·) represents a self-selection connection operation; W DGM and b DGM are the weights and biases in the convolution operation.

[0046] The present invention proposes a deep image super-resolution reconstruction method based on a convolutional neural network. Compared with the conventional deep image super-resolution reconstruction method, the present invention has the following advantages:

[0047] (1) The traditional concatenation and convolution feature fusion methods may result in insufficient feature fusion, which may lead to texture transfer in the result image. The dual feature fusion operation proposed in the present invention uses a channel attention mechanism between the depth feature, the intensity feature, and the two features, which can effectively extract the depth and intensity features and fully fuse them, thereby enhancing the guiding role of deep image reconstruction.

[0048] (2) A dual-feature guided reconstruction module is designed with multi-level guidance to fuse the depth features and intensity features to guide the super-resolution reconstruction of the depth image, thus solving the problems of texture transfer and depth loss and avoiding the problem of insufficient feature utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a network structure diagram of the present invention;

[0050] Figure 2 It is a structural diagram of a deep feature extraction branch in the feature extraction part used in the present invention;

[0051] Figure 3 It is a deep recovery and reconstruction structure diagram proposed by the present invention;

[0052] Figure 4 It is a structural diagram of the dual-channel fusion module proposed in the present invention;

[0053] Figure 5 is a schematic diagram of the channel attention operation used in the present invention;

[0054] Figure 6 It is a structural diagram of the dual-feature guided reconstruction module proposed in the present invention;

[0055] Figure 7 (a) is the real depth map of Laundry in the example;

[0056] Figure 7 (b) is the same scene color image as the Laundry depth image in the example;

[0057] Figure 7 (c) is the local real depth map of Laundry in the example;

[0058] Figure 7 (d) is the local depth map of the Laundry low-resolution depth map in the example reconstructed by the traditional JBU method;

[0059] Figure 7 (e) is the local depth map of the Laundry low-resolution depth map in the example reconstructed by the MSG neural network method;

[0060] Figure 7(f) is a local depth map of the Laundry low-resolution depth map in the example after being reconstructed by the present invention. DETAILED DESCRIPTION

[0061] The specific scheme adopted by the present invention when performing super-resolution reconstruction of a deep image with an upsampling factor of r=4 is:

[0062] (1) Dataset preparation: 92 pairs of RGB-D images were selected from the MPI Sintel depth dataset and the Middlebury dataset. The obtained image pairs were rotated and flipped by 90°, 180°, and 270°, and then the obtained depth images and color images were overlapped and sampled with a step size of 48, and cropped to obtain image blocks of size 96×96, which were used as the training set and validation set of the network; then the depth images were downsampled by 4 times one by one to obtain low-resolution depth maps of size 24×24;

[0063] The test set is 6 pairs of RGB-D images of different scenes from the Middleburry (2005) data set, and the depth map is also downsampled by 4 times as the test depth map of the present invention.

[0064] (2) Image preprocessing: The low-resolution depth map obtained in step (1) is upsampled by 4 times by bicubic interpolation to obtain a 96×96 network input depth map; for color images of the same scene, the YCrCb format is used to extract the Y channel as the intensity input image of the network.

[0065] (3) Network structure design:

[0066] The overall network structure is shown in Figure 1 ,The overall structure of the network includes two parts: feature extraction and depth ,reconstruction.

[0067] Feature extraction part reference Figure 2 , two input pyramid structures with the same structure are used to extract the depth features and intensity features of the depth image and intensity image respectively. Take the depth feature extraction as an example; the input depth image is downsampled 4 times to obtain depth maps of 48×48, 24×24, 12×12 and 6×6 in turn, and these depth maps are used as the input of each level; the input depth map of this level is convolved with a convolution kernel of 3×3, and the obtained features are concatenated with the output features of the previous level, and the output features of this level are obtained after a residual attention operation (i.e., RAM in the figure). The output of the last level contains features of multiple scales.

[0068] Deep recovery and reconstruction part reference Figure 3, the depth features and intensity features obtained at the last level in the previous part are first fused through the dual channel fusion module (DCM). The fused features are first deconvolved with a convolution kernel of 3×3 and a stride of 2×2 to enlarge the feature size and compress the channels. The enlarged features and the depth features and intensity features of the same level of the feature extraction part are put into the dual feature guided reconstruction module (DGM) for guided recovery and reconstruction. The fused features are subjected to the residual attention operation to obtain the guided features. After three cycles of operation, the reconstructed features are obtained. The final features are convolved with a convolution kernel of 1×1, and the feature channels are compressed to 1. The obtained features are pixel-added with the input depth map to obtain the reconstructed depth map.

[0069] Dual Channel Fusion Module (DCM) Reference Figure 4 , first perform channel attention operation on the two input features to enhance the features. Then the two enhanced features are first concatenated, and then a convolution operation with a convolution kernel of 1×1 is performed to complete the preliminary feature fusion. Finally, a channel attention operation is performed to obtain the fully fused features. The channel attention reference Figure 5 .

[0070] Dual-signature guided reconstruction module (DGM) reference Figure 6 First, the depth features and intensity features of the same level and size are self-selected and connected SSC to obtain the features after dual feature fusion; then they are concatenated with the output features of the previous level and then convolved with a 1×1 convolution kernel to obtain the initial fused features, and finally a channel attention operation is performed to fully fuse them to obtain the output features.

[0071] (4) Network training settings:

[0072] Use Python language and tensorflow deep learning framework to build and train deep neural network models. Figure 1 The network shown in Figure 1 is used. During training, the gap between the reconstructed image and the true depth map is calculated according to formula (1); finally, ADAM is used to optimize the network. The initial learning rate is set to 0.00001. If the loss function does not decrease within 4 epochs, the learning rate is decayed by 0.25000. If the learning rate decay is less than 10- 7 The network stops training.

[0073] (5) Evaluation index selection: This paper uses Peak Signal-to-Noise Ratio (PSNR) and Root Mean Square Error (RMSE) as evaluation indicators. The larger the PSNR value, the better the quality of the reconstructed image. The smaller the RMSE value, the closer the reconstructed depth map is to the original image, and the better the reconstruction effect.

[0074] For a comparison of the reconstruction results of the laundry depth map in the Middlebury dataset, see Figure 7 Among the other two methods, JBU is a joint bilateral upsampling method (Kopf J et al. published an article Joint bilateral upsampling), which is a traditional method; MSG is a multi-scale guidance method, which is a convolutional neural network method (Hui et al. published an article Depth map super-resolution by deep multi-scale guidance). Figure (a) is a real depth map, Figure (b) is a corresponding color map of the same scene, Figure (c) is a local depth map of the spout position extracted from the real depth map, Figure (d) is a local depth map of the same position after reconstruction by the JBU method, Figure (e) is a local depth map of the same position after reconstruction by the MSG method, and Figure (f) is a local depth map of the same position after reconstruction by the present invention.

[0075] It can be seen that the method of the present invention has good restoration in the depth discontinuous position such as the spout and the smooth position such as the background window frame and the wall. The PSNR and RMSE values ​​of the three methods are shown in the following table. It can be seen from the table that the 4× reconstruction effect of the present invention is more obvious than the other two methods.

[0076] method PSNR / dB RMSE JBU 39.70 2.64 MSG 50.18 0.79 The present invention 52.71 0.59

Claims

1. A deep image super-resolution reconstruction network based on dual feature fusion guidance, characterized in that: The training process of the entire network is as follows: (1) Dataset preparation: A certain amount of depth images and color images of the same scene are selected from different public depth image datasets. The obtained images are first rotated and flipped by 90°, 180°, and 270°. Then, the obtained depth images and color images are overlapped and sampled with a step size of 48, and cropped to obtain image blocks of size 96×96, which are used as the training set and validation set of the network. Then, the depth images are downsampled by 4 times one by one to obtain low-resolution depth maps. (2) Image preprocessing: Perform bicubic interpolation on the low-resolution depth map obtained in step (1) to make the size of the processed depth map consistent with the color map of the same scene, and obtain the depth input image of the network; then use the color map of the same scene in YCrCb format, and extract its Y channel image as the intensity input image of the network; (3) Network structure design: The network structure consists of two parts; One part is the feature extraction part, which uses two input pyramid structures with the same structure to extract the features of the input depth image and intensity image respectively; taking the depth feature extraction as an example, the input depth map is downsampled step by step, and the depth map obtained by each downsampling is first subjected to a convolution operation with a different number of channels to obtain a feature map, which is then concatenated with the features after the previous level of pooling, and then subjected to a residual attention operation to obtain the features of this layer; the features of this layer are used to guide the subsequent reconstruction, and are also used to fuse with the features of the next level to form multi-scale features. The mathematical model of this process is: Among them, i∈{1, 2, 3, 4}, is the depth image after i-times downsampling; is from The features extracted from and are the weights and biases in the convolution operation; σ(·) represents the ReLU activation function; maxpool(·) represents the downsampling operation; f RAM (·) represents the residual attention operation; represent Features obtained after residual attention operation; cat(·) represents concatenation operation; The second part is the deep recovery and reconstruction part, which mainly uses the dual-channel fusion module and the dual-feature guided reconstruction module to recover and reconstruct the extracted features; The multi-level intensity features obtained at the last level of the feature extraction part and deep features First, the dual-channel fusion module is used to obtain the fused features. The fused features are first deconvolved to enlarge the feature size and then combined with the depth features at the same level. and strength characteristics The dual-feature guided reconstruction module is jointly entered for guided recovery and reconstruction. The output features are strengthened by the residual attention feature to obtain the output features of this level. After three cycles of the above operation, the reconstructed features are obtained. Then perform a 1×1 convolution operation on the reconstructed features with a channel number of 1 and add the pixels with the input depth map to obtain the final reconstructed depth map D sr ; The mathematical model of this process is: Where j∈{0, 1, 2, 3}, f DCM (·) represents the dual-channel fusion module; f DGM (·) represents the dual-feature guided reconstruction module; deconv(·) represents deconvolution; σ represents the ReLU activation function; W sr and b sr Represent the weight and bias of the 1×1 convolution operation respectively; represents pixel addition; Among them, in the dual-channel fusion module of the second part, the input depth features and intensity features are enhanced by channel attention respectively, and then feature splicing and convolution operations are performed in sequence to complete the initial feature fusion, and then the obvious features are enhanced by channel attention again to obtain the fully fused features; the mathematical model of this module is: Among them, F d and F g They represent the depth features and intensity features obtained in the feature extraction part respectively; and They represent the depth and intensity features after channel attention enhancement; f C (·) represents the channel attention operation; W DCM and b DCM are the weights and biases in the convolution operation; Similarly, the dual-feature guided reconstruction module in the second part takes the depth features and intensity features from the feature extraction part of the same level and the reconstruction features of the previous level as input, first self-selects and connects the depth features and intensity features to obtain the fusion guide features, and then uses this to guide the reconstruction features of the previous level to perform the recovery and reconstruction operation. The mathematical model of this module is: F SSC =f SSC (F d ,F g ) (11) F mix =σ(W DGM ·cat(F SSC ,F s )+b DGM ) (12) F out =f C (F mix ) (13) Among them, f SSC (·) represents a self-selection connection operation; F SSC represents the feature obtained by self-selection connection after dual feature fusion; F s Represents the previous level reconstruction feature; F mix Representative will F SSC Features and F s The features are concatenated and convolved to obtain the initial fusion features; W DGM and b DGM are the weights and biases in the convolution operation; The entire network uses the minimum mean error (MSE) as the loss function to shorten the gap between the depth map reconstructed by the network and the real depth map: Where k represents the kth image; N represents the number of training samples each time; θ represents the parameter to be learned; represents the input depth map, Represents the Y channel image of the corresponding color image; Represents the reconstructed depth map; Represents the corresponding real depth map; (4) Network training: This network updates the network parameters through the gradient descent method based on back propagation; (5) Depth map super-resolution reconstruction: The test low-resolution depth image and the color image of the same scene are preprocessed and input into the trained network model to obtain a super-resolution depth map at the output layer.

Images