A disparity map super-resolution method based on neural network

Abstract

The present invention discloses a method for super-resolution of a disparity map based on a neural network. After data preprocessing of the input RGB, the data is input into a convolutional neural network model with residual and skip connection to obtain a disparity map with original resolution. The method proposed by the present invention is based on a neural network, with a low-resolution image as input, and a residual structure and a skip link structure are added to reduce the amount of calculation of the network model. The network has no other constraints except the loss function. Combined with the traditional algorithm, this special structure can perform high-speed and high-quality super-resolution on the disparity map without other pre-processing and post-processing methods, that is, it can break through the upper limit of the operating accuracy of the traditional algorithm and can achieve real-time operation on the mobile terminal GPU. Compared with the end-to-end neural network algorithm, under the condition of the same accuracy, the method runs faster and enhances the practicality of the stereo vision matching system.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for super-resolution of disparity maps based on a neural network. Background Art

[0002] Stereo vision matching is an important step in depth estimation, which can be used in smart production, autonomous driving, robotics and other fields. The image output by the stereo vision matching algorithm is a disparity map, which represents the change in the relative position of the same object between the left image and the right image. The distance between the target point and the pixel point can be calculated through the disparity map. At present, the mainstream ways to obtain disparity maps are mainly traditional algorithms and neural network algorithms, but they have their own problems: traditional algorithms have poor accuracy and neural network algorithms run slowly.

[0003] Resolution reduction is one of the methods to accelerate stereo vision matching. Reducing the image resolution to 1 / 4 of the original can reduce the number of stereo vision calculations to 1 / 8 of the original, greatly improving the speed of stereo vision matching calculations. However, the error of the disparity map of the original resolution obtained by the traditional interpolation method is significantly higher than that of the original algorithm, and the excessive error rate makes it difficult to meet the requirements of practical applications; disparity map enhancement is one of the means to improve the accuracy of the stereo vision matching algorithm. Using a neural network to enhance the disparity map generated by the traditional algorithm can improve the upper limit of the accuracy of the traditional algorithm. However, for high-resolution images, the amount of calculation of the neural network is too large, and there is overfitting, and the network operation speed is still difficult to meet the requirements of practical applications.

[0004] For stereo vision matching systems, it is crucial to strike a balance between speed and accuracy. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned prior art, a neural network-based disparity map super-resolution method that can run in real time on a mobile GPU is proposed.

[0006] Technical solution: A disparity map super-resolution method based on a neural network, comprising:

[0007] Step 1: Obtain a low-resolution disparity map and a low-resolution original RGB image and perform feature extraction, then splice the feature information and output it;

[0008] Step 2: Use the convolution module to reduce the dimension of the feature information output in step 1 and extract deeper feature information;

[0009] Step 3: Use the deconvolution module to increase the dimension of the feature information output in step 2 and reconstruct the feature information;

[0010] Step 4: Upsample the result obtained in step 3 together with the low-resolution disparity map and the feature information obtained in step 1 to obtain a disparity map of the original resolution.

[0011] Furthermore, in step 1, the low-resolution original RGB image is first converted into a grayscale image and then feature extraction is performed.

[0012] Furthermore, the step 2 includes the following sub-steps:

[0013] Step 2-1: Downsample the feature information obtained in step 1 through the convolution module;

[0014] Step 2-2: The result obtained in step 2-1 is further downsampled twice through the convolution module. The input of the first downsampling is the output of step 2-1, and the input of the second downsampling is the feature information extracted by the residual block after the output of the first downsampling.

[0015] Furthermore, the step 3 includes the following sub-steps:

[0016] Step 3-1: Upsample the feature information obtained in step 2-2 through a deconvolution module, where the input of the deconvolution module is the feature information extracted by the residual block from the output of step 2-2;

[0017] Step 3-2: The result obtained in step 3-1 is further upsampled twice through the deconvolution module. The input of the first upsampling is the concatenation of the output of step 3-1 and the feature information extracted by the residual block of the output of the first downsampling in step 2-2. The input of the second upsampling is the concatenation of the feature information extracted by the residual block of the output of the first upsampling and the feature information extracted by the residual block of the output of step 2-1. The output of the second upsampling is the reconstructed feature information.

[0018] Furthermore, the residual block has a single-layer neural network, and the output of the residual block is a concatenation of input information and deep feature information extracted by the single-layer neural network.

[0019] Furthermore, for all neural networks, the activation function is the ReLU function, and the loss function of the network is:

[0020]

[0021]

[0022] L=L noc +L occ

[0023] In the formula, m represents the number of valid points in the image, gt noc Represents the labeled data of the unobstructed area, gt occrepresents the labeled data of all regions, Y represents the output of the neural network, L noc represents the loss function calculated by the labeled data of the unobstructed area and the output of the neural network, L occ It represents the loss function calculated by the labeled data of all regions and the output results of the neural network. The final loss function L is the sum of the two loss functions.

[0024] Beneficial effect: The neural network-based disparity map super-resolution method of the present invention performs data preprocessing on the input RGB and then inputs it into a convolutional neural network model with residual and skip connections to obtain a disparity map with original resolution.

[0025] Specifically, the method uses low-resolution images as the input of the neural network, which reduces the amount of network calculations and improves the network's calculation speed compared to using images with original resolution as input; in addition, residual blocks are added during downsampling and upsampling, which allows the network to better process boundary information during magnification; third, during upsampling, the input of the deconvolution module is the output of the previous level and the information block of the same size obtained during the downsampling process. This skip link structure reduces the information that the network needs to learn, accelerates the convergence of the network, and reduces overfitting; fourth, when obtaining the disparity map of the original resolution, the low-resolution disparity map and the features obtained by the network are concatenated as the input of the deconvolution module, which can also accelerate network convergence and reduce overfitting.

[0026] The method proposed in the present invention is based on a neural network, with low-resolution images as input, and with residual structures and skip link structures, which reduces the computational complexity of the network model. The network has no other constraints except the loss function. Combined with the traditional algorithm, this special structure can perform high-speed and high-quality super-resolution on the disparity map without other pre-processing and post-processing methods, that is, it can break through the upper limit of the operating accuracy of the traditional algorithm and can achieve real-time operation on the mobile terminal GPU. Compared with the end-to-end neural network algorithm, under the condition of the same accuracy, this method runs faster and enhances the practicality of the stereo vision matching system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the method of the present invention;

[0028] Figure 2 is a flow chart of the method of the present invention;

[0029] Figure 3 A network structure diagram of the method of the present invention;

[0030] Figure 4 The low-resolution RGB image and disparity map in the embodiment of the present invention;

[0031] Figure 5 It is a schematic diagram of the comparison between the disparity map obtained by the method of the present invention, the annotation data and the image with the original resolution. DETAILED DESCRIPTION

[0032] The present invention will be further explained below in conjunction with the accompanying drawings.

[0033] In this embodiment, the method for super-resolution of disparity map based on neural network includes the following steps:

[0034] Step 1: Obtain a low-resolution disparity map and a low-resolution original RGB image, perform feature extraction, and output the feature information after splicing. The low-resolution original RGB image is first converted to a grayscale image before feature extraction, that is, the three channels are compressed into a single channel.

[0035] Step 2: Use the convolution module to reduce the dimension of the feature information output in step 1 and extract deeper feature information, which includes the following sub-steps:

[0036] Step 2-1: Downsample the feature information obtained in step 1 through the convolution module;

[0037] Step 2-2: The result obtained in step 2-1 is further downsampled twice through the convolution module. The input of the first downsampling is the output of step 2-1, and the input of the second downsampling is the feature information extracted by the residual block after the output of the first downsampling. The residual block is added in the downsampling process to better extract detail information and extract deeper feature information.

[0038] Step 3: Use the deconvolution module to increase the dimension of the feature information output in step 2 and reconstruct the feature information, which specifically includes the following two sub-steps:

[0039] Step 3-1: Upsample the feature information obtained in step 2-2 through a deconvolution module, where the input of the deconvolution module is the feature information extracted by the residual block from the output of step 2-2;

[0040] Step 3-2: The result obtained in step 3-1 is further upsampled twice through the deconvolution module. The input of the first upsampling is the concatenation of the output of step 3-1 and the feature information extracted by the residual block of the output of the first downsampling in step 2-2. The input of the second upsampling is the concatenation of the feature information extracted by the residual block of the output of the first upsampling and the feature information extracted by the residual block of the output of step 2-1. The output of the second upsampling is the reconstructed feature information.

[0041] Step 4: Upsample the result obtained in step 3 together with the low-resolution disparity map and the feature information obtained in step 1 to obtain a disparity map of the original resolution.

[0042] The residual block in the above steps has a single-layer neural network, and the output of the residual block is the concatenation of the input information and the deep feature information extracted by the single-layer neural network. For all the above neural networks, the activation function is the ReLU function, and the loss function of the network is:

[0043]

[0044]

[0045] L=L noc +L occ

[0046] In the formula, m represents the number of valid points in the image, gt noc Represents the labeled data of the unobstructed area, gt occ represents the labeled data of all regions, Y represents the output of the neural network, L noc represents the loss function calculated by the labeled data of the unobstructed area and the output of the neural network, L occ It represents the loss function calculated by the labeled data of all regions and the output results of the neural network. The final loss function L is the sum of the two loss functions.

[0047] The present invention uses low-resolution disparity map images for disparity map calculation, reducing the amount of stereo vision matching calculation to 1 / 8 of the original amount; by magnifying the image through a neural network, a disparity map with higher accuracy can be obtained than directly calculating the disparity map of the original image. In this embodiment, for images with a resolution of 1240*380, the system composed of a neural network and a traditional algorithm runs at a speed of 34fps on NVIDIA JetsonAGX, the error rate of the original algorithm is 6.5%, and the error rate of the output after neural network super-resolution is 5.5%, that is, under the premise of ensuring that the system runs in real time on the mobile GPU, the accuracy limit of the traditional stereo vision matching algorithm is greatly improved.

[0048] Figure 1 is a flowchart of the method of the present invention, Figure 2 is a flow chart of the method of the present invention. Figure 3 It is a structural diagram of a network model based on a neural network in an embodiment of the present invention.

[0049] Figure 4 are a low-resolution RGB image and a low-resolution disparity map in an embodiment of the present invention, Figure 4 (a) is a low-resolution RGB image. Figure 4 (b) is a low-resolution disparity map.

[0050] Figure 5is a schematic diagram of the comparison between the disparity map obtained by neural network super-resolution and the labeled data and the image with the original resolution in an embodiment of the present invention. In the figure, Figure 5 (a) is the disparity map of the labeled data. Figure 5 (b) is the disparity map calculated directly using the original resolution image through the traditional algorithm. Figure 5 (c) is the result of super-resolution through neural network.

[0051] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A disparity map super-resolution method based on a neural network, characterized in that: include: Step 1: Obtain a low-resolution disparity map and a low-resolution original RGB image and perform feature extraction, then splice the feature information and output it; Step 2: Use the convolution module to reduce the dimension of the feature information output in step 1 and extract deeper feature information; Step 3: Use the deconvolution module to increase the dimension of the feature information output in step 2 and reconstruct the feature information; Step 4: Upsample the result obtained in step 3 together with the low-resolution disparity map and the feature information obtained in step 1 to obtain a disparity map of the original resolution; The step 2 includes the following sub-steps: Step 2-1: Downsample the feature information obtained in step 1 through the convolution module; Step 2-2: The result obtained in step 2-1 is further downsampled twice through the convolution module. The input of the first downsampling is the output of step 2-1, and the input of the second downsampling is the feature information extracted by the residual block after the output of the first downsampling. The step 3 includes the following sub-steps: Step 3-1: Upsample the feature information obtained in step 2-2 through a deconvolution module, where the input of the deconvolution module is the feature information extracted by the residual block from the output of step 2-2; Step 3-2: The result obtained in step 3-1 is further upsampled twice through the deconvolution module. The input of the first upsampling is the concatenation of the output of step 3-1 and the feature information extracted by the residual block of the output of the first downsampling in step 2-2. The input of the second upsampling is the concatenation of the feature information extracted by the residual block of the output of the first upsampling and the feature information extracted by the residual block of the output of step 2-1. The output of the second upsampling is the reconstructed feature information.

2. The neural network-based disparity map super-resolution method according to claim 1, characterized in that: In step 1, the low-resolution original RGB image is first converted into a grayscale image and then feature extraction is performed.

3. The neural network-based disparity map super-resolution method according to claim 1, characterized in that: The residual block has a single-layer neural network, and the output of the residual block is the concatenation of input information and deep feature information extracted by the single-layer neural network.

4. The neural network-based disparity map super-resolution method according to claim 3, characterized in that: For all neural networks, the activation function is the ReLU function, and the loss function of the network is: L=L noc +L occ In the formula, m represents the number of valid points in the image, gt noc Represents the labeled data of the unobstructed area, gt occ represents the labeled data of all regions, Y represents the output of the neural network, L noc represents the loss function calculated by the labeled data of the unobstructed area and the output of the neural network, L occ It represents the loss function calculated by the labeled data of all regions and the output results of the neural network. The final loss function L is the sum of the two loss functions.

Images