Method and device for enhancing depth map of two-dimensional image

Abstract

The invention discloses an enhancement method and device for a depth map of a two-dimensional image, and relates to the technical field of image processing, and the method comprises the steps: carrying out the image super-resolution of a specified two-dimensional image, and obtaining a super-resolution image of the specified two-dimensional image; performing depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; the specified two-dimensional image, the super-resolution image and the depth image are input into a preset depth image enhancement model, a depth enhanced image corresponding to the specified two-dimensional image is obtained, and the depth image enhancement model is obtained by utilizing the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image and combining the contour image of the original two-dimensional image. And training the neural network model. According to the invention, on the basis of not changing the existing depth extraction model, the problems of unclear object boundary, poor foreground and background separation and the like existing in the depth image can be solved at low cost.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for enhancing a depth map of a two-dimensional image. Background Art

[0002] Monocular depth estimation refers to restoring the depth information of a 3D scene from a 2D image. This is a very important technology in the field of computer vision. Based on monocular depth estimation, it can provide effective support for downstream tasks such as 3D reconstruction, visual inspection, and autonomous driving.

[0003] The depth map extracted by the existing monocular depth estimation method has the problem of unclear object boundaries and poor foreground and background separation. In order to solve this problem, the depth extraction model is often debugged and improved through a large number of data sets to improve the accuracy of the model. However, this method requires a high cost in model improvement and training. Therefore, how to achieve low-cost improvement of the problems of unclear object boundaries and poor foreground and background separation in the depth map is an urgent problem to be solved. Summary of the invention

[0004] The embodiment of the present invention provides a method for enhancing a depth map of a two-dimensional image, which is used to enhance the depth map of the two-dimensional image, and to improve the problems of unclear object boundaries and poor foreground and background separation in the depth map at a low cost without changing the existing depth extraction model. The method includes: Performing image super-resolution on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; Performing depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; Inputting the designated two-dimensional image, the super-resolution image and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the designated two-dimensional image; The deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image.

[0005] The embodiment of the present invention further provides a device for enhancing a depth map of a two-dimensional image, which is used to enhance the depth map of the two-dimensional image, so as to improve the problems of unclear object boundaries and poor foreground and background separation in the depth map at a low cost without changing the existing depth extraction model. The device includes: A super-resolution module, used to perform image super-resolution on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; A depth extraction module, used to perform depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; A depth enhancement module, used to input the specified two-dimensional image, the super-resolution image and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image; The deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image.

[0006] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for enhancing the depth map of the two-dimensional image when executing the computer program.

[0007] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for enhancing the depth map of the two-dimensional image is implemented.

[0008] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for enhancing the depth map of the two-dimensional image is implemented.

[0009] In the embodiment of the present invention, image super-resolution is performed on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; depth extraction is performed on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; the specified two-dimensional image, the super-resolution image and the depth image are input into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image; wherein the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image. Compared with the existing method, the embodiment of the present invention does not need to change the existing depth extraction model, and uses the depth image enhancement model to input the specified two-dimensional image, the super-resolution image and the depth image of the specified two-dimensional image into the depth image enhancement model to enhance the original depth map. Moreover, the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image, and can generate a more accurate and clear depth enhanced image, thereby realizing the low-cost improvement of the problems of unclear object boundaries and poor foreground and background separation in the depth map without changing the existing depth extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] In the attached picture: Figure 1 It is a flowchart of a method for enhancing a depth map of a two-dimensional image in an embodiment of the present invention; Figure 2 is a structural diagram of a deep image enhancement model in an embodiment of the present invention; Figure 3 is a structural diagram of an improved BSRGAN network in an embodiment of the present invention; Figure 4 is a schematic diagram of a device for enhancing a depth map of a two-dimensional image in an embodiment of the present invention; Figure 5 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0013] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.

[0014] Research has found that the depth map extracted by the existing monocular depth estimation method has problems such as unclear object boundaries and poor foreground and background separation. In order to solve this problem, the depth extraction model is often debugged and improved through a large number of data sets to improve the accuracy of the model. However, this method requires a high cost for model improvement and training. Therefore, how to achieve low-cost improvement of the problems of unclear object boundaries and poor foreground and background separation in the depth map is an urgent problem to be solved.

[0015] In response to the above research, an embodiment of the present invention provides an enhancement scheme for the depth map of a two-dimensional image, which enhances the depth map of the two-dimensional image by using a depth map enhancement model optimized after the contour image of the two-dimensional image and a super-resolution image of the two-dimensional image. Without changing the existing depth extraction model, it is possible to achieve low-cost improvements in the problems of unclear object boundaries and poor foreground and background separation in the depth map.

[0016] like Figure 1 FIG. 1 is a flowchart of a method for enhancing a depth map of a two-dimensional image provided by an embodiment of the present invention. The method may include: Step 101, performing image super-resolution on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; Step 102, performing depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; Step 103, input the specified two-dimensional image, the super-resolution image and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image; wherein the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image.

[0017] In the embodiment of the present invention, image super-resolution is performed on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; depth extraction is performed on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; the specified two-dimensional image, the super-resolution image and the depth image are input into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image; wherein the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image. Compared with the existing method, the embodiment of the present invention does not need to change the existing depth extraction model, and uses the depth image enhancement model to input the specified two-dimensional image, the super-resolution image and the depth image of the specified two-dimensional image into the depth image enhancement model to enhance the original depth map. Moreover, the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image, and can generate a more accurate and clear depth enhanced image, thereby realizing the low-cost improvement of the problems of unclear object boundaries and poor foreground and background separation in the depth map without changing the existing depth extraction model.

[0018] In the embodiment of the present invention, before using the above-mentioned method for enhancing the depth map of a two-dimensional image to perform depth map enhancement, first, a depth image enhancement model needs to be trained.

[0019] The following introduces the training method of the above-mentioned deep image enhancement model.

[0020] The first step is to build training and test sets: 1.1. An original two-dimensional image (RGB image) and a real depth enhanced image of the original two-dimensional image can be obtained, wherein the real depth enhanced image of the original two-dimensional image can be a real depth information image corresponding to the original two-dimensional image obtained through observation and measurement.

[0021] 1.2. Perform contour extraction on the original two-dimensional image to obtain a contour image of the original two-dimensional image, wherein the contour extraction can be implemented based on a Sobel filter or can be extracted based on a deep learning method, which is not limited here.

[0022] In an embodiment of the present invention, a method for extracting a contour image based on an improved Sobel filter is also proposed. Specifically, first, the original two-dimensional image can be converted into a grayscale image. In the grayscale image, the value of each pixel can represent its grayscale value. The grayscale processing method can be an average method, a weighted average method, etc., which is not limited here. Then, the grayscale image of the original two-dimensional image is subjected to image filtering. The image filtering method can be a Gaussian filter, a median filter, etc., which is not limited here. Next, the gradient amplitude of each pixel in the filtered grayscale image is calculated. The gradient amplitude is used to represent the intensity of the edge at the pixel. Therefore, the image contour pixel is determined according to a preset threshold and the gradient amplitude of each pixel, that is, according to the preset threshold, each pixel is screened, and the pixel whose gradient amplitude is greater than or equal to the threshold is determined as an edge point, that is, the image contour pixel; the initial contour image is determined according to the image contour pixel, wherein the method for calculating the gradient amplitude can use a 3×3 convolution kernel, or can use convolution kernels of other sizes, or can use other methods, which are not limited here. Finally, the initial contour image is subjected to a connectivity check. For a contour image, the grayscale value of its pixel is usually one of the foreground and background values. Therefore, in an embodiment of the present invention, the grayscale value of the contour pixel that does not meet the preset connectivity condition in the initial contour image is set to the grayscale value corresponding to the background, thereby obtaining a contour image of the two-dimensional image. For example, the connectivity check process may be to traverse all contour pixel points in the filtered grayscale image, and detect in turn whether each contour pixel point has more than 30 consecutive adjacent contour pixel points (adjacent here means that the distance to this contour pixel point is 1 pixel). If this condition is met, the contour pixel point is retained, otherwise the contour pixel point is set to the grayscale value corresponding to the background. In this way, it can be ensured that the obtained contour image has less noise and is clearer.

[0023] 1.3. Perform image super-resolution on the original two-dimensional image to obtain a super-resolution image of the original two-dimensional image. The image super-resolution method will be introduced in the following embodiments and will not be described in detail here.

[0024] 1.4. Perform depth extraction on the original two-dimensional image to obtain an initial depth image of the original two-dimensional image, wherein the depth extraction method can be implemented based on an existing depth extraction model, such as existing open source methods such as zoedepth, leres, etc. The embodiment of the present invention does not limit the depth extraction method.

[0025] 1.5. Based on the original two-dimensional image, the contour image of the original two-dimensional image, the super-resolution image, the initial depth image and the real depth enhanced image, these data can be split according to a preset ratio to construct a training set and a test set.

[0026] The second step is to build the network structure of the deep image enhancement model: Figure 2 A structural diagram of a deep image enhancement model provided by an embodiment of the present invention, such as Figure 2 As shown, the deep image enhancement model may include a first feature extraction network 1, a second feature extraction network 2, a third feature extraction network 3 and a deep fusion network 4. Among them, the first feature extraction network 1 can be used to extract features from the original two-dimensional image, the second feature extraction network 2 can be used to extract features from the super-resolution image of the original two-dimensional image, the third feature extraction network 3 can be used to extract features from the initial depth image of the original two-dimensional image, and the deep fusion network 4 can be used to perform feature fusion on the outputs of the first feature extraction network 1, the second feature extraction network 2, and the third feature extraction network 3, and output a deep enhanced image.

[0027] Specifically, the first feature extraction network 1, the second feature extraction network 2, and the third feature extraction network 3 can be convolution-based neural networks, and the deep fusion network 4 can be composed of a 12-layer residual network.

[0028] The third step is to train the deep image enhancement model: Based on the deep image enhancement model built in the second step above, the training set constructed in the first step can be used to train it, and the trained deep image enhancement model can be tested using the test set.

[0029] Specifically, Figure 2 As shown, the original two-dimensional image in the training set, the contour image of the original two-dimensional image, the super-resolution image, and the initial depth image can be used as inputs of the deep image enhancement model, the first feature extraction network 1 performs feature extraction on the original two-dimensional image to obtain the first feature I, the second feature extraction network performs feature extraction on the super-resolution image to obtain the second feature X1, and the third feature extraction network 3 performs feature extraction on the initial depth image to obtain the third feature D1. Then, I, X1, and D1 are concatenated to obtain P, and P is input into the deep fusion network 4. The deep fusion network 4 outputs the predicted depth enhanced image D. In addition, in the iterative process of network training, the contour image of the input original two-dimensional image can be used to perform back propagation based on the two losses (i.e., boundary consistency loss and depth consistency loss) proposed in the embodiment of the present invention, so as to optimize the parameters of the deep fusion network 4.

[0030] The depth fusion network 4 is optimized using the contour image of the original two-dimensional image, which may specifically include the following steps: based on the contour image of the original two-dimensional image, calculating the first error of the contour pixel points on the depth enhanced image predicted by the depth fusion network; based on the contour image of the original two-dimensional image, calculating the second error of the contour pixel points on the real depth enhanced image of the original two-dimensional image; based on the first error and the second error, determining the boundary consistency loss; based on the contour image of the original two-dimensional image, calculating the third error of the non-contour pixel points on the depth enhanced image predicted by the depth fusion network and the real depth enhanced image; based on the third error, determining the depth consistency loss; based on the boundary consistency loss and the depth consistency loss, optimizing the parameters of the depth image enhancement model.

[0031] In specific implementation, the calculation steps of boundary consistency loss can be: Based on the contour image of the original two-dimensional image, all contour pixels can be selected. For each contour pixel, at the corresponding point on the depth enhanced image D predicted by the depth fusion network, the first error of each contour pixel is calculated in turn. The calculation process can be to calculate the mean square error of the depth of each contour pixel and its adjacent contour pixel (adjacent means that the pixel distance is 1. If it is adjacent to multiple other contour points, the average of all mean square errors is taken), and the depth average gradient at each contour pixel of D is obtained, that is, the first error of each contour pixel; then, based on the contour image of the original two-dimensional image, all contour pixels are selected, and at the corresponding point on the true depth enhanced image of the original two-dimensional image, the depth average gradient of the corresponding position of the true depth enhanced image is calculated, that is, the second error of each contour pixel (the calculation method is the same as the first error calculation method); then, the mean square error of the first error and the second error of all contour pixels can be used as the boundary consistency loss.

[0032] In specific implementation, the calculation steps of the depth consistency loss can be: Based on the contour image of the original two-dimensional image, all non-contour pixels can be selected, and for each non-contour pixel, the mean square error of the depth of the non-contour pixel, i.e., the third error, can be calculated in sequence at the corresponding points on the depth enhanced image D predicted by the depth fusion network and the true depth enhanced image; then, the third error can be used as the depth consistency loss.

[0033] In this way, the boundary consistency loss can be used to describe the continuity of the depth of the boundary part of the object in the image to ensure the continuity and accuracy of the overall shape of the object in the depth map. The depth consistency loss can be used to describe the image depth accuracy in the non-boundary area. The boundary consistency loss is combined with the depth consistency loss to optimize the parameters of the deep image enhancement model, thereby producing a better depth enhanced image.

[0034] It should be noted that the calculation of the first error and the second error may also select the minimum value, the maximum value, etc. of the depth difference between the current contour pixel point and the adjacent points.

[0035] It should be noted that the calculation process of the above-mentioned boundary consistency loss can not only judge the proximity based on the above-mentioned adjacent contour pixel distance of 1, but also can be adjacent to the contour pixel point in a direction obliquely 45 degrees, or can be adjacent to the contour pixel point with a distance greater than or equal to a preset number of points, or can be adjacent to the contour pixel point in a direction obliquely 45 degrees with a distance greater than or equal to 1 point, and so on.

[0036] It should be noted that the calculation process of the above-mentioned depth consistency loss can not only be based on the mean square error of the depth of non-contour pixels of the depth enhanced image predicted by the depth fusion network and the true depth enhanced image provided by the embodiment of the present invention, but also can compare the standard deviation, square difference, etc. of the non-contour pixels of the depth enhanced image predicted by the depth fusion network and the true depth enhanced image.

[0037] Based on the above three steps, a trained deep image enhancement model can be obtained.

[0038] Below Figure 1 The method for enhancing the depth map of the two-dimensional image shown is described in detail.

[0039] In an embodiment of the present invention, the input of the above-mentioned depth image enhancement model includes a two-dimensional image to be processed, a super-resolution portrait of the two-dimensional image to be processed, and an initial depth image of the two-dimensional image to be processed. Therefore, in step 101 and step 102, image super-resolution can be performed on the specified two-dimensional image (i.e., the two-dimensional image to be processed) to obtain a super-resolution image of the specified two-dimensional image; and depth extraction can be performed on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image (i.e., the initial depth image of the two-dimensional image to be processed).

[0040] In step 101, image super-resolution is performed on a specified two-dimensional image, and only a super-resolution method based on deep learning, such as SRCNN or BSRGAN network, etc., can be selected; or only a bilinear interpolation method can be selected.

[0041] In order to improve the efficiency of super-resolution and ensure the super-resolution effect, an improved image super-resolution method is provided in an embodiment of the present invention, which may specifically include the following steps: Determine the resolution of a specified 2D image; Determining a super-resolution method for the specified two-dimensional image according to the resolution of the specified two-dimensional image; According to the super-resolution method, the designated two-dimensional image is super-resolved to obtain the super-resolved image.

[0042] In one embodiment, determining a super-resolution method for a specified two-dimensional image according to a resolution of the specified two-dimensional image may include: In the case where the length or width of the resolution of the specified two-dimensional image is greater than a preset size, the super-resolution method is a bilinear interpolation method; When the length and width of the resolution of the specified two-dimensional image are both less than or equal to a preset size, the super-resolution method adopts a preset super-resolution network, wherein the super-resolution network is a multi-scale generative adversarial network architecture.

[0043] In specific implementation, the super-resolution method may include a bilinear difference method, or a method using a preset super-resolution network, wherein the super-resolution network is a multi-scale generative adversarial network architecture. Among them, the bilinear difference method has the advantages of simple calculation and fast speed, and the preset super-resolution network is a multi-scale generative adversarial network architecture, which has excellent performance and flexibility. Therefore, based on the resolution of the specified two-dimensional image, selecting a suitable super-resolution method can improve the efficiency of super-resolution while ensuring the super-resolution effect.

[0044] For example, by detecting the resolution of a specified two-dimensional image, a preset super-resolution network may be used for super-resolution for images whose length and width are both less than or equal to 512 (preset size), and a bilinear interpolation may be used for super-resolution for images whose length or width is greater than 512 (preset size).

[0045] It should be noted that the super-resolution method is not limited to the above-mentioned bilinear difference method and super-resolution network method, and may also be other super-resolution methods, which can be set according to actual needs.

[0046] In a specific implementation, the preset super-resolution network may be an existing BSRGAN network. The bilinear difference and BSRGAN networks are relatively mature technologies and will not be described in detail here.

[0047] In an embodiment of the present invention, in order to improve the accuracy of super-resolution, a method for super-resolution based on an improved BSRGAN network is provided. Specifically, the following steps may be included: when the super-resolution method adopts a preset super-resolution network, the specified two-dimensional image is input into the super-resolution network, so that the super-resolution network downsamples the specified two-dimensional image to obtain downsampled images of different scales; extracts features from each downsampled image to obtain feature maps of different scales; upsamples the feature maps of different scales to obtain multiple upsampled images; fuses the multiple upsampled images to obtain the super-resolution image, and outputs the super-resolution image.

[0048] When it comes to specific implementation, Figure 3 The structure diagram of an improved BSRGAN network provided by an embodiment of the present invention is shown in FIG. Figure 3As shown in the figure, the super-resolution network is an improved BSRGAN network. Compared with the existing BSRGAN network, the improved BSRGAN network introduces a multi-scale feature fusion mechanism, which enables the super-resolution network to better capture the different scale features of the image. Specifically, the encoder of the super-resolution network can use convolution to downsample the specified two-dimensional image to obtain downsampled images of different scales; then, multiple parallel ResNet networks (residual networks) are used to extract features from each downsampled image to obtain feature maps of different scales; then, the decoder of the super-resolution network can use convolution to upsample the feature maps of different scales to restore them to the original image size (i.e., the same size as the specified two-dimensional image) to obtain multiple upsampled images; multiple upsampled images are fused to obtain a super-resolution image, and the super-resolution image is output. In this way, by introducing a multi-scale structure, both the feature extraction performance of the encoder end and the reconstruction performance of the decoder end are enhanced, thereby obtaining a better high-resolution image.

[0049] In one embodiment, the super-resolution image of the specified two-dimensional image may be an image with a preset super-resolution multiple, for example, an image with a super-resolution multiple of 2.

[0050] It should be noted that, in the encoder of the improved BSRGAN network, convolution can be used to downsample the input features at different rates, or to upsample the input features at different rates; in the decoder of the improved BSRGAN network, convolution can be used to downsample the input features at different rates, or to upsample the input features at different rates.

[0051] In step 102, the method for performing depth extraction on the specified two-dimensional image may be implemented based on an existing depth extraction model, such as using an existing open source method, such as zoedepth, leres, etc. The embodiment of the present invention does not limit the method for depth extraction.

[0052] In the above step 103, the two-dimensional image, the super-resolution image obtained in step 101 and the depth image obtained in step 102 can be used as model inputs and input into the above-trained depth image enhancement model to obtain a depth enhanced image corresponding to the two-dimensional image.

[0053] In one embodiment, as described above, the above-mentioned deep image enhancement model may include a first feature extraction network, a second feature extraction network, a third feature extraction network and a deep fusion network.

[0054] Therefore, in the above step 103, after inputting the specified two-dimensional image, super-resolution image and depth image into the deep image enhancement model, the first feature extraction network of the deep image enhancement model performs feature extraction on the specified two-dimensional image to obtain a first depth feature image; the second feature extraction network performs feature extraction on the super-resolution image to obtain a second depth feature image; the third feature extraction network performs feature extraction on the depth image to obtain a third depth feature image; the depth fusion network generates and outputs a depth enhanced image corresponding to the specified two-dimensional image based on the first depth feature image, the second depth feature image and the third depth feature image.

[0055] In one embodiment, since the deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image, it can specifically include: optimizing the deep fusion network using the contour image of the original two-dimensional image.

[0056] In one embodiment, optimizing the deep fusion network using the contour image of the original two-dimensional image may specifically include: According to the contour image of the original two-dimensional image, the first error of the contour pixel points on the depth enhanced image predicted by the depth fusion network is calculated; According to the contour image of the original two-dimensional image, a second error of contour pixel points on the true depth enhanced image of the original two-dimensional image is calculated; Determining a boundary consistency loss based on the first error and the second error; According to the contour image of the original two-dimensional image, the third error between the non-contour pixel points on the depth enhanced image predicted by the depth fusion network and the real depth enhanced image is calculated; Determining a depth consistency loss based on the third error; The parameters of the deep image enhancement model are optimized according to the boundary consistency loss and the depth consistency loss.

[0057] The above optimization methods have been explained in detail in the model training process and will not be repeated here.

[0058] In summary, the embodiment of the present invention does not need to change the existing depth extraction model. It uses the depth image enhancement model to input the specified two-dimensional image, the super-resolution image of the specified two-dimensional image, and the depth image into the depth image enhancement model to enhance the original depth map. Moreover, the depth image enhancement model uses the original two-dimensional image, the super-resolution image of the original two-dimensional image, and the depth image, combined with the contour image of the original two-dimensional image, to train the neural network model, which can produce a more accurate and clear depth enhanced image, thereby achieving low-cost improvement of the problems of unclear object boundaries and poor foreground and background separation in the depth map without changing the existing depth extraction model. And the depth image enhancement model proposed in the embodiment of the present invention is theoretically applicable to all existing two-dimensional image depth map enhancement effects and has good generalization.

[0059] It should be noted that the method for enhancing the depth map of a two-dimensional image provided in an embodiment of the present invention can not only use the two-dimensional image to be processed, the super-resolution image of the two-dimensional image to be processed and the initial depth image corresponding to the two-dimensional image to be processed to enhance the depth map; it can also include a combination of other types of images, including but not limited to replacing the super-resolution image of the present embodiment with an image of super-resolution of different magnifications, replacing the super-resolution image of the present embodiment with an image of reduced resolution of different magnifications, replacing the initial depth image corresponding to the two-dimensional image to be processed of the present embodiment with a depth image after super-resolution of different magnifications of the two-dimensional image to be processed, replacing the initial depth of the two-dimensional image to be processed of the present embodiment with a depth image after reduced resolution of different magnifications of the two-dimensional image to be processed, replacing the initial image of the two-dimensional image to be processed of the present embodiment with other types of images after processing the two-dimensional image to be processed, and the like.

[0060] The embodiment of the present invention also provides a device for enhancing the depth map of a two-dimensional image, as described in the following embodiment. Since the principle of solving the problem by the device is similar to the above-mentioned method for enhancing the depth map of a two-dimensional image, the implementation of this method can refer to the implementation of the method for enhancing the depth map of a two-dimensional image, and the repeated parts will not be repeated.

[0061] like Figure 4 FIG. 1 is a schematic diagram of a device for enhancing a depth map of a two-dimensional image provided by an embodiment of the present invention. The device may include: The super-resolution module 401 is used to perform image super-resolution on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; A depth extraction module 402 is used to perform depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; A depth enhancement module 403 is used to input the specified two-dimensional image, the super-resolution image and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image; The deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image.

[0062] In one embodiment, the super-resolution module 401 may be specifically used for: Determine the resolution of a specified 2D image; Determining a super-resolution method for the specified two-dimensional image according to the resolution of the specified two-dimensional image; According to the super-resolution method, the designated two-dimensional image is super-resolved to obtain the super-resolved image.

[0063] In one embodiment, the super-resolution module 401 may also be used to: In the case where the length or width of the resolution of the specified two-dimensional image is greater than a preset size, the super-resolution method is a bilinear interpolation method; When the length and width of the resolution of the specified two-dimensional image are both less than or equal to a preset size, the super-resolution method adopts a preset super-resolution network, wherein the super-resolution network is a multi-scale generative adversarial network architecture.

[0064] In one embodiment, the super-resolution module 401 may also be used to: In the case where the super-resolution method adopts a preset super-resolution network, the specified two-dimensional image is input into the super-resolution network so that the super-resolution network downsamples the specified two-dimensional image to obtain downsampled images of different scales; features are extracted from each downsampled image to obtain feature maps of different scales; the feature maps of different scales are upsampled to obtain multiple upsampled images; the multiple upsampled images are fused to obtain the super-resolution image, and the super-resolution image is output.

[0065] In one embodiment, the deep image enhancement model includes a first feature extraction network, a second feature extraction network, a third feature extraction network and a deep fusion network; The depth enhancement module 403 may be specifically used for: The first feature extraction network of the deep image enhancement model extracts features from the designated two-dimensional image to obtain a first depth feature image; the second feature extraction network extracts features from the super-resolution image to obtain a second depth feature image; the third feature extraction network extracts features from the depth image to obtain a third depth feature image; the depth fusion network generates and outputs a depth enhanced image corresponding to the designated two-dimensional image based on the first depth feature image, the second depth feature image and the third depth feature image.

[0066] In one embodiment, an optimization module is also included, which can be specifically used to optimize the deep fusion network using the contour image of the original two-dimensional image.

[0067] In one embodiment, the deep optimization module may also be used to optimize the deep image enhancement model using the contour image of the original two-dimensional image: According to the contour image of the original two-dimensional image, the first error of the contour pixel points on the depth enhanced image predicted by the depth fusion network is calculated; According to the contour image of the original two-dimensional image, a second error of contour pixel points on the true depth enhanced image of the original two-dimensional image is calculated; Determining a boundary consistency loss based on the first error and the second error; According to the contour image of the original two-dimensional image, the third error between the non-contour pixel points on the depth enhanced image predicted by the depth fusion network and the real depth enhanced image is calculated; Determining a depth consistency loss based on the third error; The parameters of the deep image enhancement model are optimized according to the boundary consistency loss and the depth consistency loss.

[0068] An embodiment of the present invention further provides a computer device, Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, the method for enhancing the depth map of the above-mentioned two-dimensional image is implemented.

[0069] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for enhancing the depth map of the two-dimensional image is implemented.

[0070] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for enhancing the depth map of the two-dimensional image is implemented.

[0071] In the embodiment of the present invention, image super-resolution is performed on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; depth extraction is performed on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; the specified two-dimensional image, the super-resolution image and the depth image are input into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image, wherein the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image. Compared with the existing method, the embodiment of the present invention does not need to change the existing depth extraction model, and uses the depth image enhancement model to input the specified two-dimensional image, the super-resolution image and the depth image of the specified two-dimensional image into the depth image enhancement model to enhance the original depth map. Moreover, the depth image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image, and can generate a more accurate and clear depth enhanced image, thereby realizing that on the basis of not changing the existing depth extraction model, the problems of unclear object boundaries and poor foreground and background separation existing in the depth map can be improved at a low cost.

[0072] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0076] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for enhancing a depth map of a two-dimensional image, characterized in that: include: Performing image super-resolution on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; Performing depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; Inputting the designated two-dimensional image, the super-resolution image and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the designated two-dimensional image; The deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image.

2. The method according to claim 1, characterized in that Performing image super-resolution on the specified two-dimensional image to obtain a super-resolution image, comprising: Determine the resolution of a specified 2D image; Determining a super-resolution method for the specified two-dimensional image according to the resolution of the specified two-dimensional image; According to the super-resolution method, the designated two-dimensional image is super-resolved to obtain the super-resolved image.

3. The method according to claim 2, characterized in that Determining a super-resolution method for the specified two-dimensional image according to the resolution of the specified two-dimensional image includes: In the case where the length or width of the resolution of the specified two-dimensional image is greater than a preset size, the super-resolution method is a bilinear interpolation method; When the length and width of the resolution of the specified two-dimensional image are both less than or equal to a preset size, the super-resolution method adopts a preset super-resolution network, wherein the super-resolution network is a multi-scale generative adversarial network architecture.

4. The method according to claim 3, characterized in that According to the super-resolution method, super-resolution is performed on the specified two-dimensional image to obtain the super-resolution image, including: In the case where the super-resolution method adopts a preset super-resolution network, the specified two-dimensional image is input into the super-resolution network so that the super-resolution network downsamples the specified two-dimensional image to obtain downsampled images of different scales; features are extracted from each downsampled image to obtain feature maps of different scales; the feature maps of different scales are upsampled to obtain multiple upsampled images; the multiple upsampled images are fused to obtain the super-resolution image, and the super-resolution image is output.

5. The method according to claim 1, characterized in that The deep image enhancement model includes a first feature extraction network, a second feature extraction network, a third feature extraction network and a deep fusion network; Inputting the designated two-dimensional image, the super-resolution image, and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the designated two-dimensional image, including: The first feature extraction network of the deep image enhancement model extracts features from the designated two-dimensional image to obtain a first depth feature image; the second feature extraction network extracts features from the super-resolution image to obtain a second depth feature image; the third feature extraction network extracts features from the depth image to obtain a third depth feature image; the depth fusion network generates and outputs a depth enhanced image corresponding to the designated two-dimensional image based on the first depth feature image, the second depth feature image and the third depth feature image.

6. The method according to claim 5, characterized in that The deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image, including: optimizing the deep fusion network using the contour image of the original two-dimensional image.

7. The method according to claim 6, characterized in that The deep fusion network is optimized using the contour image of the original two-dimensional image, including: According to the contour image of the original two-dimensional image, the first error of the contour pixel points on the depth enhanced image predicted by the depth fusion network is calculated; According to the contour image of the original two-dimensional image, a second error of contour pixel points on the true depth enhanced image of the original two-dimensional image is calculated; Determining a boundary consistency loss based on the first error and the second error; According to the contour image of the original two-dimensional image, the third error between the non-contour pixel points on the depth enhanced image predicted by the depth fusion network and the real depth enhanced image is calculated; Determining a depth consistency loss based on the third error; The parameters of the deep image enhancement model are optimized according to the boundary consistency loss and the depth consistency loss.

8. A device for enhancing a depth map of a two-dimensional image, characterized in that: include: A super-resolution module, used to perform image super-resolution on a specified two-dimensional image to obtain a super-resolution image of the specified two-dimensional image; A depth extraction module, used to perform depth extraction on the specified two-dimensional image to obtain a depth image of the specified two-dimensional image; A depth enhancement module, used to input the specified two-dimensional image, the super-resolution image and the depth image into a preset depth image enhancement model to obtain a depth enhanced image corresponding to the specified two-dimensional image; The deep image enhancement model is obtained by training a neural network model using the original two-dimensional image, the super-resolution image and the depth image of the original two-dimensional image, combined with the contour image of the original two-dimensional image.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images