A method and system for depth map super-resolution with view synthesis fusion

Abstract

The application belongs to the technical field of image processing, and provides a depth map super-resolution method and system fusing view synthesis, comprising: acquiring a low-resolution depth map; obtaining a high-low resolution depth map according to the acquired low-resolution depth map and an optimized super-resolution network; in the application, a color picture of a target view point obtained by view synthesis from a high-resolution depth true value map is used as a true value of a color image; the super-resolution network is optimized by comparing the difference between the true value of the color image and the color picture of the target view point generated by the network reconstructed depth map predicted to obtain the optimized super-resolution network, the problem that the high-resolution color image is only used to extract features and fuse the features of the depth map is solved, and the precision of the super-resolution network and the effect of the depth map super-resolution are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a depth map super-resolution method and system fusing view synthesis. BACKGROUND

[0002] As a supplement to color images, depth maps have been applied to face recognition, automatic driving and three-dimensional modeling, etc. due to the provision of useful depth information. However, it is difficult to apply various computer vision tasks due to the fact that the depth map obtained by a commercial depth sensor is usually low in resolution. Therefore, depth map super-resolution, which is a method of up-sampling a low-resolution depth map to a high-resolution space, has attracted extensive attention. With the development of deep learning, depth map super-resolution can be divided into two categories according to whether a high-resolution color image is input into the network to guide the depth map super-resolution process. Since the high-resolution color image is easy to obtain and has strong structural similarity with the depth map, it can provide some guidance information in the depth map super-resolution process, and therefore the depth map super-resolution guided by the color image has achieved better results.

[0003] The inventors have found that in the method of inputting a high-resolution color image into the network to guide the depth map super-resolution process, the input of the network is a low-resolution depth map up-sampled to the same size as the high-resolution by a linear interpolation operation and a high-resolution color image. However, the high-resolution color image is only used to extract features and fuse the features of the depth map, and is not used in the loss function. Meanwhile, in depth-related tasks such as depth estimation, the color image is usually used in the loss function to guide the depth map super-resolution process. View synthesis technology is used to project the color image of the original point to the adjacent view point by the predicted depth map and the camera pose information, and the color image of the target view point is used as the true value. The difference between the color image of the target view point and the projected color image is used to measure the quality of the generated depth map. However, in the depth map super-resolution task, the data set mostly only has an image of one view point, and therefore the source view point cannot be directly projected to the target view point by using view synthesis, and the color image of the target view point and the image after projection cannot be used to compare and optimize the network. SUMMARY

[0004] To solve the above problems, the application provides a depth map super-resolution method and system fusing view synthesis. The application solves the problem that the high-resolution color image is only used to extract features and fuse the features of the depth map, improves the accuracy of the super-resolution network and the effect of the depth map super-resolution, and solves the problem that the data set mostly only has an image of one view point, and therefore the source view point cannot be directly projected to the target view point by using view synthesis, and the color image of the target view point and the image after projection cannot be used to compare and optimize the network.

[0005] In order to achieve the above object, the present application is realized by the following technical scheme:

[0006] In the first aspect, the present application provides a depth map super-resolution method of fusion view synthesis, comprising:

[0007] Obtaining a low-resolution depth map;

[0008] According to the obtained low-resolution depth map and the optimized super-resolution network, a high-low resolution depth map is obtained;

[0009] Wherein, the color image of the target view point obtained by view synthesis using the high-resolution depth true value map is used as the true value of the color image; the super-resolution network is optimized by comparing the difference between the true value of the color image and the color image of the target view point generated by the network reconstructed depth map, to obtain the optimized super-resolution network.

[0010] Further, a color image is generated using the high-resolution depth true value map and the color image as a supervision signal, the camera parameters used in the view synthesis process are randomly generated, and then the color image of the target view point is generated using the same camera parameters.

[0011] Further, the forward spatter technology is used to realize the view synthesis, and in the forward spatter, multiple source pixels contribute to a single target pixel.

[0012] Further, the calculation process of the target view point pixel value is:

[0013]

[0014] Wherein, and represent the color image of the target view point and the source view point respectively; represent the coordinates of the pixels in the source view; represent the weight values of the pixel points spattered to the surrounding multiple pixels, that is, , , represent the down and up rounding operations respectively.

[0015] Further, the reconstruction loss of the optimized super-resolution network is:

[0016]

[0017] Wherein, and are the masked images, , , is the target view point color image generated by the true depth map; The color image of the target view point is predicted.

[0018] Further, the color image of the target view point obtained through view synthesis is subjected to multiple downsampling operations, and a multi-scale reconstruction loss is calculated.

[0019] Further, when view synthesis is used, the relative poses between multiple cameras are used, and the camera poses are randomly taken in different intervals to synthesize target images of multiple different view points.

[0020] In a second aspect, the application further provides a depth map super-resolution system fusing view synthesis, comprising:

[0021] The data acquisition module is configured to acquire a low-resolution depth map.

[0022] The super-resolution module is configured to obtain a high-resolution depth map according to the acquired low-resolution depth map and the optimized super-resolution network.

[0023] The color image of the target view point obtained through view synthesis by the high-resolution depth ground truth is used as the ground truth of the color image, and the super-resolution network is optimized by comparing the difference between the ground truth of the color image and the color image of the target view point generated by the predicted network reconstructed depth map, to obtain the optimized super-resolution network.

[0024] In a third aspect, the application further provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the depth map super-resolution method fusing view synthesis of the first aspect.

[0025] In a fourth aspect, the application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the depth map super-resolution method fusing view synthesis of the first aspect when executing the program.

[0026] Compared with the prior art, the application has the following beneficial effects:

[0027] 1. In the application, the color image of the target view point obtained through view synthesis by the high-resolution depth ground truth is used as the ground truth of the color image, and the super-resolution network is optimized by comparing the difference between the ground truth of the color image and the color image of the target view point generated by the predicted network reconstructed depth map, to obtain the optimized super-resolution network, which solves the problem that the high-resolution color image is only used to extract features and fuse the features of the depth map, and improves the precision of the super-resolution network and the effect of depth map super-resolution.

[0028] 2、The high-resolution depth ground truth and color image are used to generate a color image as a supervision signal in the application, the camera parameters used in the view synthesis process are randomly generated, and then the same camera parameters are used to generate the color picture of the target view point for the depth map generated by the super-resolution network; the problem that the data set has only one view point image and cannot directly use view synthesis to project the source view point to the target view point and compare the color image of the target view point and the image after projection to optimize the network is solved;

[0029] 3、In the application, the color picture of the target view point obtained through view synthesis is subjected to multiple downsampling operations, and multi-scale reconstruction loss is calculated, and the total reconstruction loss is the combination of the reconstruction loss at each scale, thereby reducing the interference of the error depth value generated by the super-resolution network;

[0030] 4、In the application, the relative poses between multiple cameras are used in view synthesis, and the camera poses are randomly taken in different intervals to synthesize target images of multiple different view points, so that the occlusion phenomenon is better handled. BRIEF DESCRIPTION OF DRAWINGS

[0031] The drawings constituting a part of this embodiment are used to provide further understanding of the embodiment, and the illustrative embodiments of the embodiment and the description thereof are used to explain the embodiment, and do not constitute improper limitation on the embodiment.

[0032] Figure 1 The color image guided depth map super-resolution of the embodiment 1 of the application;

[0033] Figure 2 The forward sputtering process of the embodiment 1 of the application;

[0034] Figure 3 The color image as a supervision signal of the embodiment 1 of the application. DETAILED DESCRIPTION

[0035] The application will be further described below in combination with the drawings and embodiments.

[0036] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0037] Embodiment 1:

[0038] The embodiment provides a depth map super-resolution method fusing view synthesis, mainly aims at a depth map super-resolution method guided by a color image. Compared with a depth map super-resolution method not applicable to color image guidance, the super-resolution method can improve the quality of depth map reconstruction because the depth map and the color image have high structural similarity. As recorded in the background art, but the color information is only applied when the super-resolution network is trained, and only the true value depth map and the reconstructed high-resolution depth map are used to compare when the loss function of the whole network is calculated. For the above problem and at least one problem in the background art, the specific content in the embodiment is as follows:

[0039] First, view synthesis is performed. The view synthesis technique is widely used in multi-view self-supervised tasks. The depth map output by the network is used to project the color image of one viewpoint to another viewpoint combined with the camera parameters. The network is optimized by comparing the difference between the generated color image and the original image. View synthesis can be regarded as associating a pixel point from one viewpoint to the pixel value of another viewpoint. The calculation process is as follows:

[0040]

[0041] wherein, represents the coordinates of a pixel in the source view; represents the corresponding relationship from the source viewpoint to the target viewpoint; K represents the intrinsic matrix of the camera; represents the change of the relative pose between the source view and the target view, and the relative pose is represented by = , wherein R and t represent rotation and translation, respectively. Since in the data set of the depth map super-resolution task, there are mostly images of one viewpoint, in the embodiment, a depth true value map and a color image are used to generate a color image as a supervision signal. The camera parameters used in the view synthesis process are randomly generated. Then, the same camera parameters are used to generate the color image of the target viewpoint from the depth map generated by the super-resolution network. Therefore, the relative pose in the embodiment is randomly valued.

[0042] Since the pixel coordinates of the target viewpoint obtained are likely to be continuous values, in the forward mapping, linear interpolation is required on multiple coordinates near the projection point to obtain the pixel value of the pixel coordinates. For example, linear interpolation is required on four coordinates near the projection point to obtain the pixel value of the pixel coordinates. In order to solve the problem of lacking all information of the target viewpoint, in the embodiment, a forward sputtering technique is used to realize view synthesis. In the forward sputtering, each source pixel accumulates the contribution to the target pixel, as shown in Figure 2As shown; therefore, many source pixels can contribute to a single target pixel. The calculation process of the target viewpoint pixel value is as follows:

[0043]

[0044] wherein, and represent the color images of the target viewpoint and the source viewpoint respectively; represent the coordinates of the pixels in the source view; represent the weight values of the pixel points splashing to the surrounding four pixels, that is, , , represent the down and up rounding operations to take the coordinates of the four points around the projection point.

[0045] Then, the reconstruction loss based on view synthesis is performed. In the past super-resolution work, although the high-resolution color image is input into the network, the features that are helpful to the depth map super-resolution are extracted to improve the overall network performance, but when the network is optimized, only the difference between the true depth map and the predicted depth map is used, and the color image is not used, that is:

[0046]

[0047] Due to the lack of color images of the target viewpoint, after randomly obtaining the relative pose between the viewpoints, the color image of the target viewpoint obtained by view synthesis from the high-resolution depth true value map is used as the true value of the color image in this embodiment, and the network is optimized by comparing the difference between the color image generated by the predicted depth map and the color image, as shown in Figure 3 . The reconstruction error of the target viewpoint generated from the super-resolution network is calculated, that is:

[0048]

[0049] wherein, is the target viewpoint color image generated by the true depth map; is the predicted target viewpoint color image; The purpose of is to punish the color difference between and , so as to optimize the depth map super-resolution network. In the boundary area of the object, due to the discontinuity of the depth value, occlusion and holes will be generated in the projection process, and the loss calculated at the occlusion and hole positions may interfere with the training of the network, so the reconstruction loss used in this embodiment is:

[0050]

[0051] wherein, , is the masked image, wherein , , The purpose is to eliminate the influence of the voids caused by the boundary area depth value mutation of the two images on the reconstruction error, and better optimize the super-resolution network.

[0052] When the depth value generated by the super network and the true value differ too much, in the generated two images of the target view, it may not be within a pixel block, because the bilinear interpolation has gradient locality, the target is easy to fall into local minimum value, in order to reduce the interference of the super network generating error depth value, the image obtained by view synthesis is subjected to multiple downsampling operations in the embodiment, and the reconstruction loss of multiple scales is calculated, so the total reconstruction loss is the combination of the reconstruction loss at each scale. In the embodiment, two downsampling operations are performed, respectively obtaining images with original size 1 / 2 and 1 / 4, and increasing the reconstruction loss after downsampling:

[0053]

[0054] In other self-supervised depth tasks, using multiple-view images can effectively handle the influence of occlusion in the projection process on network optimization. Due to the diversity of the baseline, increasing more views can provide higher reconstruction accuracy. Therefore, in the embodiment, the relative poses between multiple cameras are used when using view synthesis, and these camera poses are randomly taken in different intervals to synthesize target images of multiple different views, and to better handle the occlusion phenomenon. In the embodiment, the average value of the multi-view reconstruction loss is used as the final reconstruction loss.

[0055]

[0056] Therefore, in the embodiment, after using the color image as the supervision signal, the loss of the overall network can be described as:

[0057]

[0058] wherein, is a hyperparameter, in the embodiment =0.5.

[0059] In order to evaluate the performance of the method in the embodiment, the performance of the model is verified on the RGB-D-D dataset. There are a total of 1449 640*480 images in the RGB-D-D dataset, according to the method of the data set author, 1000 images are selected for training in the embodiment, and the remaining 449 images are tested. The low-resolution depth map is obtained by downsampling the high-resolution depth map, and the quantitative results are shown in Table 1:

[0060] Table 1: Quantitative results of depth map super-resolution on RGB-D-D dataset

[0061]

[0062] wherein RMSE is a measurement index in centimeters, the best result is obtained by the embodiment on the RGB-D-D dataset. The RGB-D-D dataset also contains real low-resolution images, the real low-resolution images are input into the network, and the quantitative results are shown in Table 2:

[0063] Table 2: Quantitative results of depth map super-resolution on real low-resolution depth map

[0064]

[0065] Embodiment 2:

[0066] The embodiment provides a depth map super-resolution system fusing view synthesis, comprising:

[0067] The data acquisition module is configured to acquire a low-resolution depth map.

[0068] The super-resolution module is configured to obtain a high-resolution depth map according to the acquired low-resolution depth map and the optimized super-resolution network.

[0069] wherein the color image of the target view point obtained by view synthesis from the high-resolution depth ground truth map is used as the ground truth of the color image; the super-resolution network is optimized by comparing the difference between the target view point color image of the ground truth of the color image and the color image generated by the predicted network reconstructed depth map, to obtain the optimized super-resolution network.

[0070] The working method of the system is the same as the depth map super-resolution method fusing view synthesis in embodiment 1, and details are not repeated here.

[0071] Embodiment 3:

[0072] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the depth map super-resolution method fusing view synthesis in embodiment 1.

[0073] Embodiment 4:

[0074] The embodiment provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the depth map super-resolution method fusing view synthesis in embodiment 1.

[0075] The above merely provides preferred embodiments of the present embodiment, but is not intended to limit the present embodiment. For those skilled in the art, the present embodiment can have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present embodiment shall fall into the scope of protection of the present embodiment.

Claims

1. A depth map super-resolution method for fused view synthesis, characterized in that, include: Obtain a low-resolution depth map; Based on the obtained low-resolution depth map and the optimized super-resolution network, a high-resolution depth map is obtained. In this process, the color image of the target viewpoint obtained by view synthesis using a high-resolution depth ground truth map is used as the ground truth of the color image; the super-resolution network is optimized by comparing the difference between the ground truth of the color image and the color image of the target viewpoint generated by the network reconstructing the depth map, and the optimized super-resolution network is obtained. The optimized super-resolution network uses the following reconstruction loss: in, and The image after being masked. , , A target viewpoint color image generated from a true depth map; A color image of the predicted target viewpoint.

2. The depth map super-resolution method for fused view synthesis as described in claim 1, characterized in that, A color image is generated using a high-resolution depth ground truth map and a color image as a supervision signal. The camera parameters used in the view synthesis process are randomly generated. Then, the same camera parameters are used to generate a color image of the target viewpoint from the depth map generated by the super-resolution network.

3. The depth map super-resolution method for fused view synthesis as described in claim 2, characterized in that, Forward sputtering is used to achieve view composition, in which multiple source pixels contribute to a single target pixel.

4. The depth map super-resolution method for fused view synthesis as described in claim 3, characterized in that, The calculation process for the target viewpoint pixel value is as follows: in, and Color images representing the target viewpoint and the source viewpoint, respectively; Represents the coordinates of a pixel in the source view; This represents the weight value of a pixel's splash effect on multiple surrounding pixels, i.e. , , These represent rounding down and rounding up, respectively.

5. The depth map super-resolution method for fused view synthesis as described in claim 1, characterized in that, Multiple downsampling operations are performed on the color image of the target viewpoint obtained through view synthesis to calculate the reconstruction loss at multiple scales. The total reconstruction loss is a combination of the reconstruction losses at each scale.

6. The depth map super-resolution method for fused view synthesis as described in claim 1, characterized in that, When using view composition, the relative poses of multiple cameras are used, with these camera poses randomly taking values ​​in different intervals, to synthesize target images from multiple different viewpoints.

7. A depth map super-resolution system for fused view synthesis, characterized in that, include: The data acquisition module is configured to acquire low-resolution depth maps; The super-resolution module is configured to obtain a high-resolution depth map based on the acquired low-resolution depth map and the optimized super-resolution network. In this process, the color image of the target viewpoint obtained by view synthesis using a high-resolution depth ground truth map is used as the ground truth of the color image; the super-resolution network is optimized by comparing the difference between the ground truth of the color image and the color image of the target viewpoint generated by the network reconstructing the depth map, and the optimized super-resolution network is obtained. The optimized super-resolution network uses the following reconstruction loss: in, and The image after being masked. , , A target viewpoint color image generated from a true depth map; A color image of the predicted target viewpoint.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the depth map super-resolution method for fused view synthesis as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the depth map super-resolution method for fused view synthesis as described in any one of claims 1-6.

Images