Depth map extraction method and device of two-dimensional image
By extracting contours and expanding the dimensions of data from 2D images, and combining autoencoders and normal extraction modules, a clear depth map is generated, which solves the problems of unclear object boundaries and difficulty in separating foreground and background in monocular depth estimation, and improves the accuracy of the depth map.
Patent Information
- Application Number
- CN202411689532.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-01-24
AI Technical Summary
Existing monocular depth estimation methods use autoencoders, which result in smoothed object boundary regions in the depth map, unclear object boundaries, and difficulty in separating foreground and background, thus affecting the accuracy of depth map extraction.
By extracting contours from two-dimensional images and using data dimensionality expansion techniques to map the images from low-dimensional space to high-dimensional space, and combining autoencoders and normal extraction modules, depth images of multiple different scales are extracted, ultimately generating a clear depth map.
It improves the clarity of the contour boundaries and the separation of foreground and background in the depth map, ensuring the accuracy of the depth map.
Smart Images
Figure CN119360043A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for extracting a depth map of a two-dimensional image. Background Art
[0002] Monocular depth estimation refers to restoring the depth information of a 3D scene from a 2D image. This is a very important technology in the field of computer vision. Based on monocular depth estimation, it can provide effective support for downstream tasks such as 3D reconstruction, visual inspection, and autonomous driving.
[0003] Existing monocular depth estimation methods only use autoencoders. However, since the autoencoders are implemented based on convolutional neural networks, this inevitably leads to the smoothing of the object boundary areas in the extracted depth map, which in turn makes the object boundaries unclear and makes it difficult to separate the foreground and background, which greatly limits the accuracy of depth map extraction.
[0004] Therefore, how to extract a depth map with clear contours and good foreground and background separation is a problem that needs to be solved urgently. Summary of the invention
[0005] An embodiment of the present invention provides a method for extracting a depth map of a two-dimensional image, for extracting a depth map with clear contours and good foreground and background separation. The method includes: Performing contour extraction on the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; Based on the contour image and data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; Inputting the high-dimensional image data into an autoencoder to extract a plurality of first depth images of different scales; Inputting the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module to extract a first normal image; A second depth image is extracted according to the first normal image and a plurality of first depth images at different scales, where the second depth image is a final depth map.
[0006] The embodiment of the present invention further provides a depth map extraction device for a two-dimensional image, which is used to extract a depth map with clear contours and good foreground and background separation. The device includes: A contour extraction module, used for performing contour extraction on the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; A high-dimensional mapping module, used for mapping the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space based on the contour image and data dimension expansion technology, so as to obtain high-dimensional image data of the two-dimensional image to be processed; A first depth extraction module, used for inputting the high-dimensional image data into an autoencoder to extract a plurality of first depth images of different scales; A normal extraction module, used for inputting the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module to extract a first normal image; The second depth extraction module is used to extract a second depth image according to the first normal image and a plurality of first depth images of different scales, wherein the second depth image is a final depth map.
[0007] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for extracting a depth map of a two-dimensional image when executing the computer program.
[0008] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for extracting a depth map of a two-dimensional image is implemented.
[0009] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for extracting a depth map of a two-dimensional image is implemented.
[0010] In an embodiment of the present invention, contour extraction is performed on a two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; based on the contour image and data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; the high-dimensional image data is input into an autoencoder to extract a plurality of first depth images of different scales; the high-dimensional image data and the first depth images of multiple different scales are input into a preset normal extraction module to extract a first normal image; based on the first normal image and the first depth images of multiple different scales, a second depth image is extracted, and the second depth image is the final depth map. Compared with the existing monocular depth estimation method, the embodiment of the present invention extracts a depth map based on a contour map and a normal map of a two-dimensional image, which can make the contour boundary of the extracted depth map clearer, the foreground and background separation better, and ensure the accuracy of the extracted depth map. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0012] In the attached picture: Figure 1 is a flow chart of a method for extracting a depth map of a two-dimensional image in an embodiment of the present invention; Figure 2 A flow chart of a method for extracting contours of a two-dimensional image to be processed according to an embodiment of the present invention; Figure 3 This is a flow chart of a method for mapping a two-dimensional image to be processed from a low-dimensional space to a high-dimensional space in an embodiment of the present invention; Figure 4 This is an example diagram of a method for extracting a depth map of a two-dimensional image in an embodiment of the present invention; Figure 5 is a schematic diagram of a depth map extraction device for a two-dimensional image in an embodiment of the present invention; Figure 6 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION
[0013] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0014] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.
[0015] Research has found that the existing monocular depth estimation method is only implemented using autoencoders. However, since the autoencoder is implemented based on a convolutional neural network, this inevitably leads to the smoothing of the object boundary area in the extracted depth map, which in turn makes the object boundary unclear and makes it difficult to separate the foreground and background, which greatly limits the accuracy of depth map extraction.
[0016] In response to the above research, an embodiment of the present invention provides a depth map extraction solution for a two-dimensional image based on a contour map and a normal map, which can extract a depth map with clear contours and good foreground and background separation.
[0017] like Figure 1 FIG. 1 is a flowchart of a method for extracting a depth map of a two-dimensional image provided by an embodiment of the present invention. The method may include: Step 101, performing contour extraction on a two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; Step 102, based on the contour image and data dimension expansion technology, mapping the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; Step 103, inputting the high-dimensional image data into an autoencoder to extract a plurality of first depth images of different scales; Step 104, inputting the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module to extract a first normal image; Step 105 : extracting a second depth image according to the first normal image and a plurality of first depth images at different scales, wherein the second depth image is a final depth map.
[0018] In an embodiment of the present invention, contour extraction is performed on a two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; based on the contour image and data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; the high-dimensional image data is input into an autoencoder to extract a plurality of first depth images of different scales; the high-dimensional image data and the first depth images of multiple different scales are input into a preset normal extraction module to extract a first normal image; based on the first normal image and the first depth images of multiple different scales, a second depth image is extracted, and the second depth image is the final depth map. Compared with the existing monocular depth estimation method, the embodiment of the present invention extracts a depth map based on a contour map and a normal map of a two-dimensional image, which can make the contour boundary of the extracted depth map clearer, the foreground and background separation better, and ensure the accuracy of the extracted depth map.
[0019] Below Figure 1 The depth map extraction method of the two-dimensional image shown is described in detail.
[0020] In step 101, contour extraction may be performed on the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed.
[0021] In specific implementation, the two-dimensional image to be processed may be an RGB color image, and the Sobel algorithm may be used to extract the contour of the two-dimensional image to be processed, or a deep learning method may be used to extract the contour of the two-dimensional image to be processed, or other contour extraction methods may be used.
[0022] In order to extract a clearer contour image with less noise, the embodiment of the present invention improves the Sobel operator. Figure 2A flow chart of a method for extracting contours from a two-dimensional image to be processed provided by an embodiment of the present invention, such as Figure 2 As shown, the above step 101 may specifically include: Step 201, performing grayscale processing on the two-dimensional image to be processed to obtain a grayscale image; Step 202, performing image filtering on the grayscale image to obtain a filtered grayscale image; Step 203, calculating the gradient amplitude of each pixel in the filtered grayscale image, determining the image contour pixel according to a preset threshold and the gradient amplitude of each pixel, and determining the initial contour image according to the image contour pixel; Step 204, performing connectivity detection on the initial contour image, setting the grayscale values of contour pixels in the initial contour image that do not meet preset connectivity conditions to the grayscale values corresponding to the background in the initial contour image, and obtaining the contour image of the two-dimensional image to be processed.
[0023] In one embodiment, not satisfying the preset connectivity condition may include: a contour pixel point does not have a preset number of other contour pixel points within a preset range.
[0024] In the specific implementation, first, the two-dimensional image to be processed (RGB color image) can be converted into a grayscale image. In the grayscale image, the value of each pixel can represent its grayscale value; the grayscale processing method can be the average method, the weighted average method, etc., which is not limited here. Then, the grayscale image is filtered, and the image filtering method can be Gaussian filtering, median filtering, etc., which is not limited here. Next, the gradient amplitude of each pixel in the filtered grayscale image is calculated, and the gradient amplitude is used to represent the intensity of the edge at the pixel point; according to the preset threshold, each pixel point is screened, and the pixel point with a gradient amplitude greater than or equal to the threshold is determined as an edge point, that is, an image contour pixel point, and the initial contour image is determined according to the image contour pixel point, wherein the method for calculating the gradient amplitude can use a 3×3 convolution kernel, or a convolution kernel of other sizes, or other methods, which are not limited here. Finally, the initial contour image is subjected to connectivity detection. For a contour image, the grayscale value of its pixel point is usually one of the two values of foreground and background. Therefore, in the embodiment of the present invention, the grayscale value of the contour pixel point that does not meet the preset connectivity condition in the initial contour image is set to the grayscale value corresponding to the background, thereby obtaining the contour image of the two-dimensional image to be processed. For example, the connectivity detection process can be to traverse all contour pixel points in the initial contour image, and sequentially detect whether each contour pixel point has more than 30 adjacent continuous contour pixel points (here, adjacent means that the distance from this contour pixel point is 1 pixel or 2 pixels, or other values within the distance of 1-2 pixels). If this condition is met, the contour pixel point is retained, otherwise the contour pixel point is set to the grayscale value corresponding to the background. In this way, it can be ensured that the obtained contour image has less noise and is clearer.
[0025] In step 102, based on the contour image and data dimension expansion technology, the two-dimensional image to be processed can be mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed.
[0026] In specific implementation, the position encoding algorithm can be used for data dimension expansion. Therefore, the position encoding algorithm can be used to map the pixel coordinates of the two-dimensional image to be processed to a high-dimensional space to obtain high-dimensional image data; other data dimension expansion algorithms can also be used to map the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space.
[0027] In order to improve the subsequent processing speed, the position encoding algorithm is improved in the embodiment of the present invention. Figure 3 A flowchart of a method for mapping a two-dimensional image to be processed from a low-dimensional space to a high-dimensional space is provided in an embodiment of the present invention, such as Figure 3 As shown, the above step 102 may specifically include: Step 301, determining the image complexity according to the number of all pixels in the contour image and the number of contour pixels; Step 302, determining the sampling frequency of the image position coding according to the image complexity and the corresponding relationship between the preset sampling frequency and the image complexity; Step 303, performing Fourier transform according to the sampling frequency of the image position code, mapping each pixel point in the two-dimensional image to be processed from the low-dimensional space to the high-dimensional space, and obtaining high-dimensional image data of the two-dimensional image to be processed.
[0028] In one embodiment, the above step 301 may specifically include: The image complexity is calculated using the following formula: 00% (1) Among them, S represents the image complexity, N represents the number of all pixels in the contour image, and N L Indicates the number of contour pixels in the contour image.
[0029] In one embodiment, in the above step 302, the corresponding relationship between the preset sampling frequency and the image complexity can be expressed by the following relationship: (2) in, represents the sampling frequency, and S represents the image complexity.
[0030] In specific implementation, the sampling frequency of the image position encoding corresponding to the image complexity can be determined by the above formula (2): .
[0031] In one embodiment, the above step 303 may specifically include: Determining a mapping dimension according to a sampling frequency of the position code; According to the Fourier transform formula and the mapping dimension, each pixel point in the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed.
[0032] The above step 303 is described in detail below with a specific example: First, the sampling frequency of the image position encoding can be , determine the mapping dimension, for example, the mapping dimension can be ( ) dimension. Secondly, for each pixel in the two-dimensional image to be processed, assume that it is a five-tuple ,in, are the horizontal and vertical coordinates corresponding to the pixel point, R, G, and B are the color value components of the pixel point respectively; Mapping ,in, is the Fourier transform corresponding to x ( )-dimensional vector, is the Fourier transform corresponding to y ( )-dimensional vector. Specifically, the Fourier transform formula can be: (3) (4) In this way, the final That is, the high-dimensional image data of the two-dimensional image to be processed, whose dimension is (3+4×( )) dimension. Through the above improved position encoding algorithm, the dimension of the output data is greatly reduced, which is more conducive to improving the speed of subsequent processing.
[0033] In step 103, the high-dimensional image data obtained in step 102 may be input into an autoencoder to extract a plurality of first depth images of different scales.
[0034] In specific implementation, the autoencoder can be implemented based on a convolutional neural network, or based on other methods such as ResNet or transformer. The high-dimensional image data obtained in step 102 is input into the autoencoder, and each layer of the decoder of the autoencoder will output depth maps of different sizes, that is, multiple first depth images of different scales. For example, the autoencoder can output a first depth image whose length and width are both 1 / 4 of the size of the two-dimensional image to be processed, a first depth image whose length and width are both 1 / 2 of the size of the two-dimensional image to be processed, a first depth image of the same size as the two-dimensional image to be processed, and so on.
[0035] It should be noted that in the above step 103, any variation of the autoencoder may be used, which may be an autoencoder encoder of any depth, an autoencoder decoder of any depth, or other forms of depth map extraction methods may be used.
[0036] In the above step 104, the high-dimensional image data and a plurality of first depth images of different scales may be input into a preset normal extraction module to extract a first normal image.
[0037] In an embodiment of the present invention, the above-mentioned normal extraction module may include a first normal extraction unit, a second normal extraction unit and a normal fusion unit, wherein the first normal extraction unit is used to extract a normal image from a plurality of first depth images of different scales; the second normal extraction unit is used to extract a normal image from high-dimensional image data; and the normal fusion unit is used to fuse the depth image extracted by the first normal extraction unit and the depth image extracted by the second normal extraction unit to output a final normal image.
[0038] In specific implementation, a first normal extraction unit extracts a normal image from first depth images of multiple different scales, a second normal extraction unit extracts a normal image from high-dimensional image data, and a normal fusion unit fuses the normal images output by the two units to obtain a final normal image, i.e., a first normal image.
[0039] It should be noted that the above-mentioned normal extraction module can also be a module of other structures, such as a model based on a transformer network, an autoencoder network, or a residual network, which is not limited here.
[0040] In one embodiment, in order to improve the accuracy of the output of the normal extraction module, before the above step 104, the following steps may be further included: Upsampling a plurality of first depth images of different scales to obtain a plurality of third depth images; Performing image fusion on the multiple third depth images to obtain a fourth depth image; The above step 104 may specifically include: The fourth depth image and the high-dimensional image data are input into a preset normal extraction module to extract a first normal image; wherein the normal extraction module includes a first normal extraction unit, a second normal extraction unit and a normal fusion unit, the first normal extraction unit extracts the second normal image according to the fourth depth image; the second normal extraction unit extracts the third normal image according to the high-dimensional image data; the normal fusion unit fuses the second normal image and the third normal image to obtain the first normal image.
[0041] In specific implementation, each first depth image can be upsampled, and the scale of the first depth image can be restored to the same size as the two-dimensional image to be processed; the upsampled multiple depth images (third depth images) are image fused to obtain a fourth depth image. Specifically, the first normal extraction unit extracts the second normal image based on the fourth depth image; the second normal extraction unit extracts the third normal image based on the high-dimensional image data; the normal fusion unit fuses the second normal image and the third normal image to obtain the first normal image. Then, the fourth depth image and the high-dimensional image data are used as inputs of the normal extraction module.
[0042] It should be noted that if the scale of the first depth map is the same as that of the two-dimensional image to be processed, it may be chosen not to be upsampled.
[0043] In the above step 105, a second depth image may be extracted according to the first normal image and a plurality of first depth images at different scales, and the second depth image is the final depth map.
[0044] In a specific implementation, the first depth feature extraction module can be used to extract depth features from the first normal image, and the second depth feature extraction module can be used to extract depth features from the first depth image, and the depth features extracted by the two can be fused to generate a final depth map. The first depth feature extraction module and the second depth feature extraction module can be modules designed based on existing neural networks, or can be autoencoders, etc.
[0045] In an embodiment of the present invention, a model training method for a normal extraction module is also provided, which may specifically include the following steps: Constructing a training set and a test set according to the original two-dimensional image, the contour image corresponding to the original two-dimensional image, the high-dimensional image data, the first depth images and the first normal images of multiple different scales; Use the training set to train the normal extraction module; The normal extraction module is tested using the test set.
[0046] In one embodiment, in order to generate a normal map with better effect, the above model training method may further include: Calculate the first error of the contour pixel points on the normal image output by the normal extraction module according to the contour image corresponding to the original two-dimensional image; Calculating, according to the contour image corresponding to the original two-dimensional image, a second error of the contour pixel point on the first normal image corresponding to the original two-dimensional image; Calculate, according to the contour image corresponding to the original two-dimensional image, a third error of the non-contour pixel point on the first normal image corresponding to the original two-dimensional image and the normal image output by the normal extraction module; determining a geometric loss based on the first error and the second error; According to the third error, determining the normal loss; According to the geometric loss and normal loss, the parameters of the normal extraction module are optimized.
[0047] During specific implementation, the embodiment of the present invention designs constraints for the normal map prediction process, that is, a new loss function called geometric consistency loss and normal consistency loss is designed for the training process of the normal extraction module. Specifically, first, based on the contour image corresponding to each original two-dimensional image, all contour pixel points can be selected in turn, and the first error between each contour point and its adjacent contour point can be calculated in turn on the normal image output by the normal extraction module (i.e., the predicted normal image) (adjacent can refer to a pixel distance of 1, and if it is adjacent to multiple other contour pixel points, the average of all errors is taken); then, at the contour pixel point corresponding to the first normal image corresponding to the original two-dimensional image (here the first normal image can be understood as the real normal image corresponding to the original two-dimensional image), the second error is calculated, and the calculation process of the second error is the same as that of the first error; and according to the first normal image corresponding to the original two-dimensional image and the normal image output by the normal extraction module, the third error of the non-contour pixel points on the image is calculated; finally, the difference between the first error and the second error can be determined as the geometric loss, and the third error can be determined as the normal loss. According to the geometric loss and the normal loss, the parameters of the normal extraction module are optimized, so that the first normal image output by the normal extraction module has a better effect.
[0048] The first error, the second error and the third error may be mean square errors, or may be the minimum value, the maximum value, etc. of the difference between the current pixel point and the adjacent points.
[0049] In order to more clearly understand the above-mentioned method for extracting the depth map of the two-dimensional image, the following is an explanation of the above-mentioned method for extracting the depth map of the two-dimensional image with a specific example. Figure 4 , Figure 4 An example diagram of a method for extracting a depth map of a two-dimensional image provided by an embodiment of the present invention.
[0050] like Figure 4As shown, first, the contour of the two-dimensional image to be processed is extracted to obtain a contour image I; then, according to the contour image I and the improved position encoding algorithm of the present invention, the two-dimensional image to be processed is position encoded, that is, each pixel point of the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space, and a high-dimensional image data P of the two-dimensional image to be processed is obtained; then, the high-dimensional image data P is input into the autoencoder to extract a plurality of first depth images of different scales, that is, S4, S2, S1, S4 are first depth images with a length and width of 1 / 4 of the size of the two-dimensional image to be processed Figure, S2 is the first depth map with a length and width of 1 / 2 the size of the two-dimensional image to be processed, and S1 is the first depth map output by the autoencoder with the same size as the two-dimensional image to be processed; S4, S2, S1 are input into the normal extraction module, and the high-dimensional image data P is input into the normal extraction module, and the normal F2 is extracted according to S4, S2, S1, and the normal F1 is extracted according to the high-dimensional image data P, and F1 and F2 are fused to obtain the first normal image F3 of the normal information; finally, the depth feature extraction is performed on the first normal image F3 to obtain the depth information S F , extract the depth features of S4, S2, and S1 to obtain the depth information S C , the depth information S F and depth information S C After fusion, the final depth image S is obtained.
[0051] In addition, in the embodiment of the present invention, Figure 4 The implementation process of the depth map extraction method of the two-dimensional image is encapsulated into a depth map extraction model. During the model training process, the depth map output by the model can be compared with the depth map of the real two-dimensional image to obtain the depth loss. Based on the depth loss, geometric loss and normal loss, the depth map extraction model is optimized and trained.
[0052] In summary, in an embodiment of the present invention, contour extraction is performed on a two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; based on the contour image and data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; the high-dimensional image data is input into an autoencoder to extract a plurality of first depth images of different scales; the high-dimensional image data and the first depth images of multiple different scales are input into a preset normal extraction module to extract a first normal image; based on the first normal image and the first depth images of multiple different scales, a second depth image is extracted, and the second depth image is the final depth map. Compared with the existing monocular depth estimation method, the embodiment of the present invention extracts a depth map based on a contour map and a normal map of a two-dimensional image, which can make the contour boundary of the extracted depth map clearer, the foreground and background separation better, and ensure the accuracy of the extracted depth map.
[0053] The present invention also provides a two-dimensional image depth map extraction device in an embodiment, as described in the following embodiment. Since the principle of solving the problem by the device is similar to the above-mentioned two-dimensional image depth map extraction method, the implementation of this method can refer to the implementation of the two-dimensional image depth map extraction method, and the repeated parts will not be repeated.
[0054] like Figure 5 FIG. 1 is a schematic diagram of a depth map extraction device for a two-dimensional image provided by an embodiment of the present invention. The device may include the following steps: The contour extraction module 501 is used to extract the contour of the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; A high-dimensional mapping module 502 is used to map the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space based on the contour image and data dimension expansion technology to obtain high-dimensional image data of the two-dimensional image to be processed; A first depth extraction module 503, configured to input the high-dimensional image data into an autoencoder to extract a plurality of first depth images of different scales; A normal extraction module 504, configured to input the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module to extract a first normal image; The second depth extraction module 505 is configured to extract a second depth image according to the first normal image and a plurality of first depth images of different scales, wherein the second depth image is a final depth map.
[0055] In one embodiment, the contour extraction module 501 may be specifically used for: Performing grayscale processing on the two-dimensional image to be processed to obtain a grayscale image; Performing image filtering on the grayscale image to obtain a filtered grayscale image; Calculating the gradient amplitude of each pixel in the filtered grayscale image, determining the image contour pixel according to a preset threshold and the gradient amplitude of each pixel, and determining the initial contour image according to the image contour pixel; Connectivity detection is performed on the initial contour image, and the grayscale values of contour pixels in the initial contour image that do not meet preset connectivity conditions are set to the grayscale values corresponding to the background in the initial contour image to obtain the contour image of the two-dimensional image to be processed.
[0056] In one embodiment, not satisfying the preset connectivity condition may include: a contour pixel point does not have a preset number of other contour pixel points within a preset range.
[0057] In one embodiment, the high-dimensional mapping module 502 may be specifically used for: Determining image complexity according to the number of all pixels in the contour image and the number of contour pixels; Determining the sampling frequency of the image position coding according to the image complexity and the corresponding relationship between the preset sampling frequency and the image complexity; According to the sampling frequency of the image position code, Fourier transform is performed to map each pixel point in the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed.
[0058] In one embodiment, the high-dimensional mapping module 502 may also be used to: The image complexity is calculated using the following formula: 00% Among them, S represents the image complexity, N represents the number of all pixels in the contour image, and N L Indicates the number of contour pixels in the contour image.
[0059] In one embodiment, the corresponding relationship between the preset sampling frequency and the image complexity can be expressed as: in, represents the sampling frequency, and S represents the image complexity.
[0060] In one embodiment, the high-dimensional mapping module 502 may also be used to: Determining a mapping dimension according to a sampling frequency of the image position encoding; According to the Fourier transform formula and the mapping dimension, each pixel point in the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed.
[0061] In one embodiment, the normal extraction module may include a first normal extraction unit, a second normal extraction unit and a normal fusion unit, wherein the first normal extraction unit is used to extract a normal image from a plurality of first depth images of different scales; the second normal extraction unit is used to extract a normal image from high-dimensional image data; and the normal fusion unit is used to perform image fusion on the depth image extracted by the first normal extraction unit and the depth image extracted by the second normal extraction unit.
[0062] In one embodiment, an upsampling module may also be included, which is used for, before the normal extraction module inputs the high-dimensional image data and the first depth images of multiple different scales into a preset normal extraction module to extract the first normal image: Upsampling a plurality of first depth images of different scales to obtain a plurality of third depth images; Performing image fusion on the multiple third depth images to obtain a fourth depth image; Normal extraction module, which can be used for: The fourth depth image and the high-dimensional image data are input into a preset normal extraction module to extract a first normal image; wherein a first normal extraction unit of the normal extraction module extracts a second normal image based on the fourth depth image; a second normal extraction unit extracts a third normal image based on the high-dimensional image data; and a normal fusion unit fuses the second normal image and the third normal image to obtain the first normal image.
[0063] In one embodiment, a model training module may also be included, which is used for, before the normal extraction module inputs the high-dimensional image data and the first depth images of multiple different scales into a preset normal extraction module and extracts the first normal image: Train the normal extraction module through the following steps: Constructing a training set and a test set according to the original two-dimensional image, the contour image corresponding to the original two-dimensional image, the high-dimensional image data, the first depth images and the first normal images of multiple different scales; Use the training set to train the normal extraction module; The normal extraction module is tested using the test set.
[0064] In one embodiment, the model training module may also be used to: Calculate the first error of the contour pixel point on the normal image output by the normal extraction module according to the contour image corresponding to the original two-dimensional image; Calculating, according to the contour image corresponding to the original two-dimensional image, a second error of the contour pixel point on the first normal image corresponding to the original two-dimensional image; Calculate, according to the contour image corresponding to the original two-dimensional image, a third error of the non-contour pixel point on the first normal image corresponding to the original two-dimensional image and the normal image output by the normal extraction module; determining a geometric loss based on the first error and the second error; According to the third error, determining the normal loss; According to the geometric loss and normal loss, the parameters of the normal extraction module are optimized.
[0065] An embodiment of the present invention further provides a computer device, Figure 6 Schematic diagram of a computer device in an embodiment of the present invention, the computer device 600 includes a memory 610, a processor 620, and a computer program 630 stored in the memory 610 and executable on the processor 620, and the processor 620 implements the above-mentioned two-dimensional image depth map extraction method when executing the computer program 630.
[0066] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for extracting a depth map of a two-dimensional image is implemented.
[0067] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned depth map extraction method of a two-dimensional image is implemented.
[0068] In an embodiment of the present invention, contour extraction is performed on a two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; based on the contour image and data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; the high-dimensional image data is input into an autoencoder to extract a plurality of first depth images of different scales; the high-dimensional image data and the first depth images of multiple different scales are input into a preset normal extraction module to extract a first normal image; based on the first normal image and the first depth images of multiple different scales, a second depth image is extracted, and the second depth image is the final depth map. Compared with the existing monocular depth estimation method, the embodiment of the present invention extracts a depth map based on a contour map and a normal map of a two-dimensional image, which can make the contour boundary of the extracted depth map clearer, the foreground and background separation better, and ensure the accuracy of the extracted depth map.
[0069] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0070] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0071] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0073] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for extracting a depth map of a two-dimensional image, characterized in that: include: Performing contour extraction on the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; Based on the contour image and data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed; Inputting the high-dimensional image data into an autoencoder to extract a plurality of first depth images of different scales; Inputting the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module to extract a first normal image; A second depth image is extracted according to the first normal image and a plurality of first depth images at different scales, where the second depth image is a final depth map.
2. The method according to claim 1, characterized in that Performing contour extraction on the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed includes: Performing grayscale processing on the two-dimensional image to be processed to obtain a grayscale image; Performing image filtering on the grayscale image to obtain a filtered grayscale image; Calculating the gradient amplitude of each pixel in the filtered grayscale image, determining the image contour pixel according to a preset threshold and the gradient amplitude of each pixel, and determining the initial contour image according to the image contour pixel; Performing connectivity detection on the initial contour image, setting the grayscale values of contour pixels in the initial contour image that do not meet a preset connectivity condition to the grayscale values corresponding to the background in the initial contour image, and obtaining a contour image of the two-dimensional image to be processed; Wherein, not satisfying the preset connectivity condition includes: a contour pixel point does not have a preset number of other contour pixel points within a preset range.
3. The method according to claim 1, characterized in that Based on the contour image and the data dimension expansion technology, the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed, including: Determining image complexity according to the number of all pixels in the contour image and the number of contour pixels; Determining the sampling frequency of the image position coding according to the image complexity and the corresponding relationship between the preset sampling frequency and the image complexity; According to the sampling frequency of the image position code, Fourier transform is performed to map each pixel point in the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed.
4. The method according to claim 3, characterized in that According to the sampling frequency of the image position code, Fourier transform is performed to map each pixel point in the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed, including: Determining a mapping dimension according to a sampling frequency of the image position encoding; According to the Fourier transform formula and the mapping dimension, each pixel point in the two-dimensional image to be processed is mapped from a low-dimensional space to a high-dimensional space to obtain high-dimensional image data of the two-dimensional image to be processed.
5. The method according to claim 1, characterized in that The normal extraction module includes a first normal extraction unit, a second normal extraction unit and a normal fusion unit, wherein the first normal extraction unit is used to extract a normal image from a plurality of first depth images of different scales; the second normal extraction unit is used to extract a normal image from high-dimensional image data; and the normal fusion unit is used to perform image fusion on the depth image extracted by the first normal extraction unit and the depth image extracted by the second normal extraction unit.
6. The method according to claim 1, characterized in that Inputting the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module, before extracting the first normal image, further comprising: Train the normal extraction module through the following steps: Constructing a training set and a test set according to the original two-dimensional image, the contour image corresponding to the original two-dimensional image, the high-dimensional image data, the first depth images and the first normal images of multiple different scales; Use the training set to train the normal extraction module; The normal extraction module is tested using the test set.
7. The method according to claim 6, characterized in that Also includes: Calculate the first error of the contour pixel points on the normal image output by the normal extraction module according to the contour image corresponding to the original two-dimensional image; Calculating, according to the contour image corresponding to the original two-dimensional image, a second error of the contour pixel point on the first normal image corresponding to the original two-dimensional image; Calculate, according to the contour image corresponding to the original two-dimensional image, a third error of the non-contour pixel point on the first normal image corresponding to the original two-dimensional image and the normal image output by the normal extraction module; determining a geometric loss based on the first error and the second error; According to the third error, determining the normal loss; According to the geometric loss and normal loss, the parameters of the normal extraction module are optimized.
8. A device for extracting a depth map of a two-dimensional image, characterized in that: include: A contour extraction module, used for performing contour extraction on the two-dimensional image to be processed to obtain a contour image of the two-dimensional image to be processed; A high-dimensional mapping module, used for mapping the two-dimensional image to be processed from a low-dimensional space to a high-dimensional space based on the contour image and data dimension expansion technology, so as to obtain high-dimensional image data of the two-dimensional image to be processed; A first depth extraction module, used for inputting the high-dimensional image data into an autoencoder to extract a plurality of first depth images of different scales; A normal extraction module, used for inputting the high-dimensional image data and a plurality of first depth images of different scales into a preset normal extraction module to extract a first normal image; The second depth extraction module is used to extract a second depth image according to the first normal image and a plurality of first depth images of different scales, wherein the second depth image is a final depth map.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.