Image depth estimation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image depth estimation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an image to be subjected to depth estimation, and extracting a pixel point feature of each pixel point in the image and a plane feature of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used for representing a plane in a view finding space of the image; a first distance is formed between a plane in a view finding space of the image and a camera shooting the image; determining a second distance between each pixel point and each depth plane through the depth estimation model according to the pixel point feature of each pixel point and the plane feature of each depth plane; and through the depth estimation model, according to a second distance between each pixel point and each depth plane, determining a depth value of each pixel point, and obtaining a depth map corresponding to the image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image depth estimation method, device, electronic device and storage medium. Background Art

[0002] Depth estimation of an image refers to estimating the depth value of each pixel in the image. The depth value of a pixel can be understood as the distance between the object represented by the pixel and the camera that captured the image. Currently, depth estimation of an image can be performed using a supervised model or a self-supervised model. However, after the existing depth estimation methods perform depth estimation on an image, the estimated depth value has a large error and the accuracy of the depth estimation is low. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, device, electronic device and storage medium for image depth estimation, which can improve the accuracy of depth estimation for images.

[0004] In a first aspect, an embodiment of the present disclosure provides an image depth estimation method, comprising:

[0005] Acquire an image for which depth estimation is to be performed, and extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; and there is a first distance between the plane in the viewing space of the image and a camera that captures the image;

[0006] Determine, by the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes;

[0007] The depth estimation model is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane, so as to obtain a depth map corresponding to the image.

[0008] In a second aspect, an embodiment of the present disclosure provides an image depth estimation device, including:

[0009] A feature extraction unit, used to obtain an image to be depth estimated, and extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; the plane in the viewing space of the image has a first distance from a camera that captures the image;

[0010] a distance determination unit, configured to determine, by using the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes;

[0011] The result determination unit is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane through the depth estimation model, so as to obtain a depth map corresponding to the image.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0014] In one or more embodiments of the present disclosure, an image to be depth estimated is obtained, and a pre-trained depth estimation model is used to extract the pixel features of each pixel in the image and the plane features of each depth plane in the image, the depth plane is used to represent the plane in the framing space of the image, and the plane in the framing space of the image has a first distance from the camera that captures the image, and the second distance between each pixel and each depth plane is determined by the depth estimation model according to the pixel features of each pixel and the plane features of each depth plane, and the depth value of each pixel is determined by the depth estimation model according to the second distance between each pixel and each depth plane, so as to obtain a depth map corresponding to the image. It can be seen that through this embodiment, the plane in the framing space of the image can be simulated by using the depth plane, and the depth value of each pixel can be determined by determining the relative distance between each pixel in the image and each depth plane, thereby improving the accuracy of depth estimation of the image and generating a high-precision depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0016] Figure 1A schematic diagram of a flow chart of an image depth estimation method provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of a depth estimation principle of a depth estimation model is provided for an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of a training process of a depth estimation model provided in an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of the training principle of a depth estimation model provided in one embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of the structure of an image depth estimation device provided by an embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0023] The embodiments of the present disclosure provide a method, device, electronic device and storage medium for image depth estimation, which can improve the accuracy of depth estimation for images.

[0024] Figure 1 A flowchart of an image depth estimation method provided by an embodiment of the present disclosure is shown. The method can be applied to a server and executed by the server, such as Figure 1 As shown, the process includes:

[0025] Step S102, obtaining an image for which depth estimation is to be performed, and extracting pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; and there is a first distance between the plane in the viewing space of the image and a camera that captures the image;

[0026] Step S104, determining a second distance between each pixel and each depth plane according to a pixel feature of each pixel and a plane feature of each depth plane through a depth estimation model;

[0027] Step S106: Determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane through the depth estimation model, and obtain a depth map corresponding to the image.

[0028] In this embodiment, an image to be depth estimated is obtained, and a pre-trained depth estimation model is used to extract the pixel feature of each pixel in the image and the plane feature of each depth plane in the image. The depth plane is used to represent the plane in the framing space of the image. The plane in the framing space of the image has a first distance from the camera that captures the image. The depth estimation model is used to determine the second distance between each pixel and each depth plane according to the pixel feature of each pixel and the plane feature of each depth plane. The depth estimation model is used to determine the depth value of each pixel according to the second distance between each pixel and each depth plane, and a depth map corresponding to the image is obtained. It can be seen that through this embodiment, the plane in the framing space of the image can be simulated by using the depth plane, and the depth value of each pixel can be determined by determining the relative distance between each pixel in the image and each depth plane, thereby improving the accuracy of depth estimation of the image and generating a high-precision depth map.

[0029] In the above step S102, an image to be depth estimated is obtained. The image may be an RGB (red, green, blue) image, and the content of the image captured is not limited in this implementation. A depth estimation model is pre-trained, and the depth estimation model can perform depth estimation on the image to obtain a corresponding depth map. The depth map is an image in which pixels correspond to pixels in the above image one by one, and the pixel value of each pixel in the depth map represents the depth value of each corresponding pixel in the above image.

[0030] In this step, the above image is input into a pre-trained depth estimation model, and the pixel features of each pixel in the image and the plane features of each depth plane in the image are extracted through the pre-trained depth estimation model. In this embodiment, it is assumed that there are multiple depth planes in the image, and the depth plane in the image is used to represent the plane in the image's framing space, that is, the depth plane is the representation of the plane in the image's framing space in the image. The image's framing space refers to the space being photographed. For example, if the image is an image taken of a park, the framing space is the park. In the image's framing space, multiple planes are included, and the depth plane corresponds one-to-one to the plane in the image's framing space. There is a first distance between the plane in the image's framing space and the camera that captures the image. Both the plane in the image's framing space and the depth plane can be planes in the vertical direction.

[0031] The principle of determining the depth value of each pixel in the image in the present embodiment is as follows: based on the distance from each point in the framing space to each plane in the framing space, the distance from each point in the framing space to the camera that captures the image can be determined, and the distance from each point in the framing space to the camera that captures the image is the depth value of each point in the framing space, that is, the depth value of each pixel in the image. Therefore, the depth plane is obtained by simulating the plane in the framing space in the image, and the depth value of each pixel in the image can be determined by calculating the distance between the pixel in the image and the depth plane in the image.

[0032] In one embodiment, extracting pixel features of each pixel in the image and plane features of each depth plane in the image includes:

[0033] Perform feature extraction on the image to obtain the pixel features of each pixel in the image;

[0034] According to the pixel features of each pixel, the plane features of each depth plane in the image are extracted.

[0035] Figure 2 A schematic diagram of a depth estimation principle of a depth estimation model is provided for an embodiment of the present disclosure, such as Figure 2 As shown, the depth estimation model includes a point feature extraction unit and a plane feature extraction unit. The image is input into the depth estimation model, and the point feature extraction unit can be used to extract features of the image to obtain the pixel features of each pixel in the image. Next, the pixel features of each pixel in the image are input into the plane feature extraction unit, and the plane feature extraction unit can extract the plane features of each depth plane in the image based on the pixel features of each pixel. Among them, the point feature extraction unit can extract features of the image and output a feature map, and the feature map represents the pixel features of each pixel in the image. In the feature map, the features of each pixel are the pixel features of each pixel in the image. The plane features of each depth plane in the image can be represented by vectors, and the vectors correspond one to one to the plane features.

[0036] In one example, the point feature extraction unit may be a neural network including a residual connection network skip-connections, for example, a U-net network including a residual connection network skip-connections. The neural network including the residual connection network skip-connections can extract the pixel features of each pixel in the image and retain high-frequency features, such as retaining the edge features of objects in the image. The plane feature extraction unit may be a shallow visual encoder (Vision Transformer, ViT), which may implicitly model the depth plane in the image based on the pixel features of each pixel in the image to obtain the plane features of each depth plane in the image. When implicitly modeling the depth plane in the image, the visual encoder may first divide the feature map output by the point feature extraction unit into multiple blocks, for example, into 16 blocks, and then add position encoding to each block of the image to obtain the plane features of each depth plane in the image.

[0037] It can be seen that through this embodiment, it is possible to perform feature extraction on the image to obtain pixel features of each pixel in the image, and based on the pixel features of each pixel, extract the plane features of each depth plane in the image, which has the effect of efficiently and accurately extracting pixel features and plane features.

[0038] In the above step S104, the second distance between each pixel and each depth plane is determined by the depth estimation model according to the pixel feature of each pixel and the plane feature of each depth plane.

[0039] In one embodiment, determining the second distance between each pixel and each depth plane according to the pixel feature of each pixel and the plane feature of each depth plane includes:

[0040] Dot product processing is performed on the pixel feature of each pixel and the plane feature of each depth plane to obtain a second distance between each pixel and each depth plane.

[0041] like Figure 2 As shown, the depth estimation model includes a relative distance codec unit, and the pixel features of each pixel and the plane features of each depth plane are input into the relative distance codec unit. The pixel features of each pixel and the plane features of each depth plane are subjected to dot product processing by the relative distance codec unit. The dot product result is the distance between the pixel and the depth plane, and the distance is used as the second distance. It can be understood that the number of dot product results is equal to the number of pixels multiplied by the number of depth planes. The dot product results can be stored in the self-cost volume.

[0042] like Figure 2As shown, each square represents a pixel point and corresponds to the second distance between the pixel point and a depth plane. Figure 2 Starting from each depth plane, the second distance between each pixel point and the depth plane is illustrated for each depth plane. Figure 2 In this case, n is a positive integer.

[0043] It can be seen that through this embodiment, dot product processing can be performed on the pixel feature of each pixel and the plane feature of each depth plane, and the second distance between each pixel and each depth plane can be accurately obtained through the dot product method.

[0044] In the above step S106, the depth value of each pixel is determined according to the second distance between each pixel and each depth plane through the depth estimation model, so as to obtain a depth map corresponding to the image.

[0045] In one embodiment, determining the depth value of each pixel point according to the second distance between each pixel point and each depth plane includes:

[0046] Determine, according to a second distance between each pixel and each depth plane, an estimated depth value of each pixel relative to each depth plane and a depth value weight of each depth plane;

[0047] The depth value of each pixel is determined according to the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane.

[0048] In this embodiment, first, the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane are determined according to the second distance between each pixel and each depth plane. Then, the depth value of each pixel is determined according to the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane.

[0049] It can be seen that through this embodiment, the depth value of each pixel can be accurately determined by determining the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane, thereby improving the accuracy of determining the depth value.

[0050] In one embodiment, determining an estimated depth value of each pixel relative to each depth plane and a depth value weight of each depth plane according to a second distance between each pixel and each depth plane includes:

[0051] For each pixel point, normalize the second distance between the pixel point and each depth plane, and determine each second distance after the normalization as an estimated depth value of the pixel point relative to each depth plane;

[0052] For each depth plane, a normalization process is performed on the second distance between each pixel point and the depth plane, and a depth value weight of the depth plane is determined according to each second distance after the normalization process.

[0053] In this embodiment, for each pixel, the second distance between the pixel and each depth plane is normalized, and each second distance after normalization is determined as the estimated depth value of the pixel relative to each depth plane. In one example, there are three depth planes, and the second distances between pixel A and each depth plane are X1, X2, and X3, respectively. X1, X2, and X3 are normalized, respectively, and each second distance after normalization is the estimated depth value of pixel A relative to each depth plane. Similarly, the estimated depth value of each pixel relative to each depth plane is determined. The estimated depth value of each pixel relative to each depth plane can also be called the depth distribution of each pixel between each depth plane. Among them, the normalization process can be processed based on the normalized exponential function softmax function.

[0054] like Figure 2 As shown, in this embodiment, each square represents a pixel point and corresponds to a second distance between the pixel point and a depth plane. Figure 2 Starting from each depth plane, the second distance between each pixel and the depth plane is illustrated. On this basis, for each depth plane, the square blocks corresponding to the depth plane can be arranged according to the position of the pixel in the image to obtain Figure 2 The squares in the image correspond to the depth planes one by one, and the squares in each square correspond to the second distance between each pixel in the corresponding image and the corresponding depth plane. Figure 2 , for the bold squares in the figure, each bold square represents the same pixel in the image, and the second distance between the pixel represented by the square and each depth plane is normalized, and each normalized second distance is determined as the estimated depth value of the pixel relative to each depth plane. Figure 2 , for each pixel point, the second distance between the pixel point and each depth plane is normalized, and each second distance after normalization is determined as the estimated depth value of the pixel point relative to each depth plane. This process can also be called the process of normalizing each second distance corresponding to the pixel point between depth planes.

[0055] In this embodiment, for each depth plane, the second distance between each pixel point and the depth plane is normalized, and the depth value weight of the depth plane is determined according to each second distance after normalization. In one example, there are three depth planes and nine pixels. For depth plane B, the second distances between each pixel point and the depth plane B are Y1, Y2, Y3...Y9, respectively. Y1, Y2, Y3...Y9 are normalized, and the depth value weight of the depth plane B is determined according to each second distance after normalization. And so on, the depth value weight of each depth plane is determined. Among them, the normalization process can be processed based on the normalized exponential function softmax function.

[0056] It can be seen that through this embodiment, the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane can be accurately determined, and then the depth value of each pixel can be accurately determined.

[0057] In one embodiment, determining the depth value weight of the depth plane according to each second distance after normalization includes:

[0058] Multiplying each second distance after normalization by the corresponding second distance before normalization;

[0059] Linearly map each multiplication result to the depth value weight of the depth plane.

[0060] In an example, there are three depth planes and nine pixel points. For depth plane B, the second distances between each pixel point and the depth plane B are Y1, Y2, Y3...Y9, respectively. Y1, Y2, Y3...Y9 are normalized, and each second distance after normalization is multiplied by the corresponding second distances before normalization, that is, Y1 after normalization is multiplied by Y1, and Y2 after normalization is multiplied by Y2, so as to obtain nine multiplication results. Then, the nine multiplication results are linearly mapped to the depth value weight of depth plane B. The depth value weight of depth plane B can be a value between 0-1, such as 0.5.

[0061] refer to Figure 2 , each square represents a pixel and corresponds to the second distance between the pixel and a depth plane, Figure 2Starting from each depth plane, the second distance between each pixel and the depth plane is illustrated for each depth plane. On this basis, for each depth plane, the second distance between each pixel and the depth plane is normalized, and each second distance after normalization is multiplied by the second distance before normalization of the corresponding pixel. For each depth plane, it corresponds to the multiplication result of each pixel in the image, and the multiplication result of all pixels can be represented by a vector. The vector includes the multiplication result of all pixels in the image. Therefore, each depth plane corresponds to a vector, which can be called a counting vector. The counting vectors of all depth planes are spliced ​​to obtain a splicing vector, and the splicing vector is input into the multilayer perceptron (MLP) in the relative distance encoding and decoding unit for linear mapping. Each counting vector is mapped to the depth value weight of the corresponding depth plane to obtain the depth value weight of each depth plane, and the depth value weight can also be called a depth bin Depthbins. The depth value weight of each depth plane can be a value between 0 and 1.

[0062] It can be seen that through this embodiment, each second distance after normalization can be multiplied with the corresponding second distance before normalization, and each multiplication result can be linearly mapped to the depth value weight of the depth plane, so as to accurately determine the depth value weight of each depth plane.

[0063] In one embodiment, determining the depth value of each pixel point according to the estimated depth value of each pixel point relative to each depth plane and the depth value weight of each depth plane includes:

[0064] For each pixel, the estimated depth value of the pixel relative to each depth plane is weighted and summed according to the depth value weight of each depth plane to obtain the depth value of the pixel.

[0065] Through the previous process, the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane can be determined. Among them, each pixel has an estimated depth value relative to each depth plane, and each depth plane has a depth value weight. Therefore, in this embodiment, for each pixel, according to the depth value weight of each depth plane, the estimated depth value of the pixel relative to each depth plane is weighted and summed to obtain the depth value of the pixel. For example, there are three depth planes and nine pixels. For pixel C, it has estimated depth values ​​C1, C2, and C3 relative to each depth plane, and the depth value weights of each depth plane are Z1, Z2, and Z3 respectively. For pixel C, its depth value is equal to = C1*Z1+C2*Z2+C3*Z3. By analogy, the depth value of each pixel is determined in this way.

[0066] like Figure 2 As shown, for each pixel, the estimated depth value of the pixel relative to each depth plane can be weighted summed according to the depth value weight of each depth plane to obtain the depth value of the pixel. It can be seen that through this embodiment, the depth value of each pixel can be obtained efficiently and quickly through weighted summation.

[0067] In this embodiment, a depth map corresponding to the image may be generated according to the depth value of each pixel in the image.

[0068] The above introduces the depth estimation process based on the depth estimation model. The following introduces the training process of the above depth estimation model.

[0069] Figure 3 A schematic diagram of the training process of a depth estimation model provided in an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the training process includes:

[0070] Step S302, obtaining a sample image and a reference image for model training, and predicting a sample depth map corresponding to the sample image through a pre-built neural network structure;

[0071] Step S304, reconstructing the sample image according to the camera motion data between the sample image and the reference image, the reference image and the sample depth map;

[0072] Step S306: Based on the sample image, the reconstructed sample image and the sample depth map, a neural network structure is trained to obtain a depth estimation model.

[0073] Figure 4 A schematic diagram of the training principle of a depth estimation model provided by an embodiment of the present disclosure. Figure 4As shown, the sample image and reference image for model training are obtained, and the sample depth map corresponding to the sample image is predicted through the pre-built neural network structure. Then, the sample image is reconstructed according to the camera motion data between the sample image and the reference image, the reference image and the sample depth map. Finally, based on the sample image, the reconstructed sample image and the sample depth map, the neural network structure is trained in a self-supervised manner to obtain a depth estimation model.

[0074] It can be seen that through this embodiment, the sample image can be reconstructed during the model training process, and the neural network structure can be trained in a self-supervised manner based on the sample image, the reconstructed sample image and the sample depth map to obtain a depth estimation model. There is no need to label the sample data during the training process, which improves the training efficiency and can generate a high-precision depth estimation model, thereby improving the accuracy of depth estimation of the image.

[0075] In the above step S302, sample images and reference images for model training are obtained. In one embodiment, monocular video data is obtained, and sample images and reference images are selected from the monocular video data. The sample images and reference images are video frames that do not contain objects moving relative to the ground, and the sample images and reference images are separated by a preset number of frames, such as 6 frames. It can be understood that since the sample image and the reference image are derived from the same monocular video data, the sample image and the reference image correspond to the same framing space and contain the content of the same framing space.

[0076] In the above step S302, a sample depth map corresponding to the sample image is predicted through a pre-built neural network structure. The prediction process is similar to the aforementioned process of generating a depth map, which will be described in detail later.

[0077] In the above step S304, reference Figure 4 , the camera motion data between the sample image and the reference image is determined through the motion data determination network, the camera motion data including the relative translation matrix and relative rotation matrix between the sample image and the reference image of the camera recording the above monocular video data. The input of the motion data determination network includes the sample image and the reference image, and the output includes the camera motion data between the sample image and the reference image. And, if Figure 4 As shown, the sample image is reconstructed through an image reconstruction network according to the camera motion data between the sample image and the reference image, the reference image and the predicted sample depth map. The image reconstruction network can be a wrap network. The input of the image reconstruction network includes the camera motion data between the sample image and the reference image, the reference image and the sample depth map, and the output includes the reconstructed sample image.

[0078] In the above step S306, based on the sample image, the reconstructed sample image and the sample depth map, the neural network structure is trained in a self-supervised manner to obtain a depth estimation model. The self-supervised training method can learn a rich scene geometry prior, such as human body priors in various postures, geometric priors of various furniture, etc., so as to train a high-precision depth estimation model and improve the accuracy of depth estimation of the image.

[0079] In one embodiment, a sample depth map corresponding to a sample image is predicted by using a pre-built neural network structure, including:

[0080] By using a neural network structure, pixel features of each sample pixel in the sample image and plane features of each sample depth plane in the sample image are extracted; the sample depth plane is used to represent a plane in the viewing space of the sample image; there is a third distance between the plane in the viewing space of the sample image and the camera that shoots the sample image;

[0081] Through the neural network structure, the sample depth map corresponding to the sample image is predicted according to the pixel features of each sample pixel and the plane features of each sample depth plane.

[0082] In this embodiment, it is assumed that there are multiple sample depth planes in the sample image, and the sample depth plane in the sample image is used to represent the plane in the framing space of the sample image, that is, the sample depth plane is the representation of the plane in the framing space of the sample image in the sample image. The framing space of the sample image refers to the photographed space. For example, if the sample image is an image taken of a park, then the framing space is the park. The framing space of the sample image includes multiple planes, and the sample depth plane corresponds one-to-one to the plane in the framing space of the sample image. There is a third distance between the plane in the framing space of the sample image and the camera that shoots the sample image. Both the plane in the framing space of the sample image and the sample depth plane can be planes in the vertical direction.

[0083] refer to Figure 4 , through the point feature extraction unit in the neural network structure, the sample image can be feature extracted to obtain the pixel feature of each sample pixel in the sample image. Then, the pixel feature of each sample pixel in the sample image is input into the plane feature extraction unit, and the plane feature extraction unit can extract the plane feature of each sample depth plane in the sample image according to the pixel feature of each sample pixel. The specific explanation of the point feature extraction unit and the plane feature extraction unit can refer to the previous description, which will not be repeated here.

[0084] Among them, the point feature extraction unit can perform feature extraction on the sample image and output a sample feature map, which represents the pixel feature of each sample pixel in the sample image. In the sample feature map, the feature of each pixel is the pixel feature of each sample pixel in the sample image. The plane feature of each sample depth plane in the sample image can be represented by a vector, and the vector corresponds to the plane feature of the sample depth plane one by one. Then, based on the pixel feature of each sample pixel and the plane feature of each sample depth plane, the sample depth map corresponding to the sample image is predicted.

[0085] It can be seen that through this embodiment, by setting the sample depth plane, based on the pixel features of each sample pixel and the plane features of each sample depth plane, the efficiency of predicting the sample depth map corresponding to the sample image can be improved.

[0086] In one embodiment, predicting a sample depth map corresponding to a sample image according to a pixel feature of each sample pixel and a plane feature of each sample depth plane includes:

[0087] Determine a fourth distance between each sample pixel and each sample depth plane according to a pixel feature of each sample pixel and a plane feature of each sample depth plane;

[0088] According to the fourth distance between each sample pixel and each sample depth plane, the depth value of each sample pixel is determined to obtain a sample depth map.

[0089] In this embodiment, first, the relative distance between each sample pixel and each sample depth plane is determined according to the pixel features of each sample pixel and the plane features of each sample depth plane, and the relative distance is the fourth distance. Then, the depth value of each sample pixel is determined according to the fourth distance between each sample pixel and each sample depth plane to obtain a sample depth map.

[0090] The principle of determining the depth value of each sample pixel in the sample image in the present embodiment is as follows: based on the distance from each point in the framing space of the sample image to each plane in the framing space of the sample image, the distance between each point in the framing space of the sample image and the camera that shoots the sample image can be determined, and the distance between each point in the framing space of the sample image and the camera that shoots the sample image is the depth value of each point in the framing space of the sample image, that is, the depth value of each sample pixel in the sample image. Therefore, the sample depth plane is obtained by simulating the plane in the framing space in the sample image, and the depth value of each sample pixel in the sample image can be determined by calculating the relative distance between the sample pixel in the sample image and the sample depth plane in the sample image.

[0091] It can be seen that, through this embodiment, the depth value of each sample pixel point can be accurately determined based on the fourth distance between each sample pixel point in the sample image and each sample depth plane, so as to generate a sample depth map.

[0092] In one embodiment, determining a fourth distance between each sample pixel and each sample depth plane according to a pixel feature of each sample pixel and a plane feature of each sample depth plane includes:

[0093] Dot product processing is performed on the pixel feature of each sample pixel and the plane feature of each sample depth plane to obtain a fourth distance between each sample pixel and each sample depth plane.

[0094] like Figure 4 As shown, the neural network structure includes a relative distance codec unit, and the pixel features of each sample pixel and the plane features of each sample depth plane are input into the relative distance codec unit. The pixel features of each sample pixel and the plane features of each sample depth plane are subjected to dot product processing by the relative distance codec unit. The dot product result is the distance between the sample pixel and the sample depth plane, and the distance is used as the fourth distance. It can be understood that the number of dot product results is equal to the number of sample pixels multiplied by the number of sample depth planes. The dot product results can be stored in the self-cost volume.

[0095] like Figure 4 As shown, each square represents a sample pixel point and corresponds to the fourth distance between the sample pixel point and a sample depth plane. Figure 4 Starting from each sample depth plane, the fourth distance between each sample pixel and the sample depth plane is illustrated for each sample depth plane.

[0096] It can be seen that through this embodiment, dot product processing can be performed on the pixel feature of each sample pixel and the plane feature of each sample depth plane, and the fourth distance between each sample pixel and each sample depth plane can be accurately obtained by dot product.

[0097] In one embodiment, determining the depth value of each sample pixel point according to a fourth distance between each sample pixel point and each sample depth plane includes:

[0098] Determine, according to a fourth distance between each sample pixel and each sample depth plane, an estimated depth value of each sample pixel relative to each sample depth plane and a depth value weight of each sample depth plane;

[0099] The depth value of each sample pixel is determined according to the estimated depth value of each sample pixel relative to each sample depth plane and the depth value weight of each sample depth plane.

[0100] In this embodiment, first, the estimated depth value of each sample pixel relative to each sample depth plane and the depth value weight of each sample depth plane are determined according to the fourth distance between each sample pixel and each sample depth plane. Then, the depth value of each sample pixel is determined according to the estimated depth value of each sample pixel relative to each sample depth plane and the depth value weight of each sample depth plane.

[0101] It can be seen that through this embodiment, the depth value of each sample pixel point can be accurately determined by determining the estimated depth value of each sample pixel point relative to each sample depth plane and the depth value weight of each sample depth plane, thereby improving the accuracy of determining the depth value.

[0102] In one embodiment, determining the estimated depth value of each sample pixel point relative to each sample depth plane and the depth value weight of each sample depth plane according to the fourth distance between each sample pixel point and each sample depth plane includes:

[0103] For each sample pixel point, normalize the fourth distance between the sample pixel point and each sample depth plane, and determine each fourth distance after the normalization as an estimated depth value of the sample pixel point relative to each sample depth plane;

[0104] For each sample depth plane, a normalization process is performed on the fourth distance between each sample pixel point and the sample depth plane, and a depth value weight of the sample depth plane is determined according to each fourth distance after the normalization process.

[0105] In this embodiment, for each sample pixel, the fourth distance between the sample pixel and each sample depth plane is normalized, and each fourth distance after normalization is determined as the estimated depth value of the sample pixel relative to each sample depth plane. In one example, there are three sample depth planes, and the fourth distances between the sample pixel T and each sample depth plane are T1, T2, and T3, respectively. T1, T2, and T3 are normalized, respectively, and each fourth distance after normalization is the estimated depth value of the sample pixel T relative to each sample depth plane. Similarly, the estimated depth value of each sample pixel relative to each sample depth plane is determined. The estimated depth value of each sample pixel relative to each sample depth plane can also be called the depth distribution of each sample pixel between each sample depth plane. Among them, the normalization process can be processed based on the normalized exponential function softmax function.

[0106] like Figure 4 As shown, in this embodiment, each square represents a sample pixel point and corresponds to the fourth distance between the sample pixel point and a sample depth plane. Figure 4 Starting from each sample depth plane, the fourth distance between each sample pixel and the sample depth plane is illustrated. On this basis, for each sample depth plane, the square blocks corresponding to the sample depth plane can be arranged according to the positions of the sample pixel points in the sample image to obtain Figure 4 The squares in the sample depth plane correspond one to one, and the squares in each square correspond to the fourth distance between each sample pixel in the sample image and the corresponding sample depth plane. Figure 4 , for the bold squares in the figure, each of which represents the same sample pixel in the sample image, the fourth distance between the sample pixel represented by the square and each sample depth plane is normalized, and each fourth distance after normalization is determined as the estimated depth value of the sample pixel relative to each sample depth plane. Figure 4 , for each sample pixel, the fourth distance between the sample pixel and each sample depth plane is normalized, and each fourth distance after normalization is determined as the estimated depth value of the sample pixel relative to each sample depth plane. This process can also be called the process of normalizing each fourth distance corresponding to the sample pixel between the sample depth planes.

[0107] In this embodiment, for each sample depth plane, the fourth distance between each sample pixel and the sample depth plane is normalized, and the depth value weight of the sample depth plane is determined according to each fourth distance after normalization. In one example, there are three sample depth planes and nine sample pixels. For the sample depth plane P, the fourth distances between each sample pixel and the sample depth plane P are P1, P2, P3...P9, respectively. P1, P2, P3...P9 are normalized, and the depth value weight of the sample depth plane P is determined according to each fourth distance after normalization. And so on, the depth value weight of each sample depth plane is determined. Among them, the normalization process can be processed based on the normalized exponential function softmax function.

[0108] It can be seen that through this embodiment, the estimated depth value of each sample pixel relative to each sample depth plane and the depth value weight of each sample depth plane can be accurately determined, and then the depth value of each sample pixel can be accurately determined.

[0109] In one embodiment, determining the depth value weight of the sample depth plane according to each fourth distance after normalization includes:

[0110] multiplying each fourth distance after normalization processing by the corresponding fourth distance before normalization processing;

[0111] Linearly map each multiplication result to a depth value weight for the sample depth plane.

[0112] In an example, there are three sample depth planes and nine sample pixel points. For the sample depth plane P, the fourth distances between each sample pixel point and the sample depth plane P are P1, P2, P3...P9, respectively. P1, P2, P3...P9 are normalized, and each fourth distance after normalization is multiplied by the corresponding fourth distances before normalization, that is, P1 after normalization is multiplied by P1, and P2 after normalization is multiplied by P2, so as to obtain nine multiplication results. Then, the nine multiplication results are linearly mapped to the depth value weight of the sample depth plane P. The depth value weight of the sample depth plane P can be a value between 0-1, such as 0.5.

[0113] refer to Figure 4 , each square represents a sample pixel and corresponds to the fourth distance between the sample pixel and a sample depth plane, Figure 4 Starting from each sample depth plane, the fourth distance between each sample pixel and the sample depth plane is illustrated for each sample depth plane. On this basis, for each sample depth plane, the fourth distance between each sample pixel and the sample depth plane is normalized, and each fourth distance after normalization is multiplied by the fourth distance before normalization of the corresponding sample pixel. For each sample depth plane, it corresponds to the multiplication result of each sample pixel in the sample image, and the multiplication result of all pixels can be represented by a vector, which includes the multiplication result of all sample pixels in the sample image. Therefore, each sample depth plane corresponds to a vector, which can be called a count vector. The count vectors of all sample depth planes are spliced ​​to obtain a spliced ​​vector, and the spliced ​​vector is input into the multilayer perceptron (MLP) in the relative distance encoding and decoding unit for linear mapping. Each count vector is mapped to the depth value weight of the corresponding sample depth plane to obtain the depth value weight of each sample depth plane. The depth value weight can also be called depth bins. The depth value weight of each sample depth plane can be a value between 0 and 1.

[0114] It can be seen that through this embodiment, each fourth distance after normalization can be multiplied with the corresponding fourth distance before normalization, and each multiplication result can be linearly mapped to the depth value weight of the sample depth plane, so as to accurately determine the depth value weight of each sample depth plane.

[0115] In one embodiment, determining the depth value of each sample pixel point according to the estimated depth value of each sample pixel point relative to each sample depth plane and the depth value weight of each sample depth plane includes:

[0116] For each sample pixel, according to the depth value weight of each sample depth plane, a weighted sum is performed on the estimated depth value of the sample pixel relative to each sample depth plane to obtain the depth value of the sample pixel.

[0117] Through the previous process, the estimated depth value of each sample pixel relative to each sample depth plane and the depth value weight of each sample depth plane can be determined. Among them, each sample pixel has an estimated depth value relative to each sample depth plane, and each sample depth plane has a depth value weight. Therefore, in this embodiment, for each sample pixel, according to the depth value weight of each sample depth plane, the estimated depth value of the sample pixel relative to each sample depth plane is weighted and summed to obtain the depth value of the sample pixel. For example, there are three sample depth planes and nine sample pixels. For the sample pixel U, it has estimated depth values ​​U1, U2, and U3 relative to each sample depth plane, and the depth value weights of each sample depth plane are N1, N2, and N3 respectively. For the pixel U, its depth value is equal to =U1*N1+U2*N2+U3*N3. By analogy, the depth value of each sample pixel is determined in this way.

[0118] like Figure 4 As shown, for each sample pixel, the estimated depth value of the sample pixel relative to each sample depth plane can be weighted summed according to the depth value weight of each sample depth plane to obtain the depth value of the sample pixel. It can be seen that through this embodiment, the depth value of each sample pixel can be obtained efficiently and quickly through weighted summation.

[0119] In this embodiment, a sample depth map corresponding to the sample image can be generated according to the depth value of each sample pixel in the sample image. Moreover, after reconstructing the sample image, a neural network structure can be trained based on the sample image, the reconstructed sample image and the sample depth map to obtain a depth estimation model.

[0120] In one embodiment, the neural network structure can be trained by the following three loss functions to obtain a depth estimation model.

[0121]

[0122] Formula (1) is the photometric loss, which is a photometric reconstruction loss function. It is used to compare the reconstructed sample image with the sample image and measure the similarity between the reconstructed sample image and the sample image at the RGB level. a represents the sample image, I b represents the reconstructed sample image, SSIM (Structural Similarity) is a standard function for calculating the similarity between images, and α is a coefficient, which can be 0.5.

[0123]

[0124] Formula (2) is Smooth loss, which is a regularization function used to make the predicted sample depth map smoother and emphasize the depth gradient of the edge of the object. In formula (2), represents the sample depth map, I t Represents the sample image. Formula (2) can make the depth of the edge part of the object in the sample depth map change greatly and the depth of the non-edge part of the object change smoothly by taking partial derivatives of the sample depth map and the sample image in the x direction and the y direction, so that the depth of each object is continuous.

[0125]

[0126] Formula (3) is Geometric loss, which is a constraint function for scene geometric consistency. In formula (3), is the reconstructed sample depth map, D' b (p) is the predicted sample depth map.

[0127] Specifically, the sample depth map corresponding to the sample image can be predicted through the previous process. Based on the above-mentioned image reconstruction principle, the reference depth map corresponding to the reference image can be obtained, and the sample depth map can be reconstructed according to the camera motion data between the sample image and the reference image, the reference depth map and the sample depth map, so as to obtain the above-mentioned Based on the reconstructed sample depth map and the predicted sample depth map, the scene geometric consistency in the depth estimation process can be constrained.

[0128] Through the above three loss functions, the training efficiency of the depth estimation model can be improved, and the depth estimation model can perform depth estimation with high precision, improve the accuracy of depth estimation of images, and generate high-precision depth maps.

[0129] The images used in the above embodiments can all be monocular images. Therefore, through the above method flow, in order to solve the problem of low accuracy of depth estimation of monocular images, depth estimation can be performed by determining the relative distance between pixels and depth planes. On the one hand, the geometric clues in the monocular image are fully utilized to avoid direct end-to-end prediction of depth. On the other hand, the detailed features of the pixels and the global features of the depth plane can be taken into account, thereby improving the accuracy of depth estimation. In addition, the generalization of depth estimation can be improved. The trained depth estimation model can be applied to depth estimation in a variety of scenarios with or without fine-tuning.

[0130] Figure 5 A schematic diagram of the structure of an image depth estimation device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the device comprises:

[0131] A feature extraction unit 51 is used to obtain an image to be depth estimated, and to extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; and there is a first distance between the plane in the viewing space of the image and a camera that captures the image;

[0132] a distance determining unit 52, configured to determine, by using the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes;

[0133] The result determination unit 53 is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane through the depth estimation model, so as to obtain a depth map corresponding to the image.

[0134] Optionally, the feature extraction unit 51 is specifically used for:

[0135] Performing feature extraction on the image to obtain pixel features of each pixel in the image;

[0136] The plane features of each depth plane in the image are extracted according to the pixel features of each pixel.

[0137] Optionally, the distance determining unit 52 is specifically configured to:

[0138] Dot product processing is performed on the pixel feature of each pixel and the plane feature of each depth plane to obtain a second distance between each pixel and each depth plane.

[0139] Optionally, the result determination unit 53 is specifically configured to:

[0140] Determining, according to a second distance between each pixel and each depth plane, an estimated depth value of each pixel relative to each depth plane and a depth value weight of each depth plane;

[0141] The depth value of each pixel is determined according to the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane.

[0142] Optionally, the result determination unit 53 is further specifically configured to:

[0143] For each of the pixel points, normalize the second distance between the pixel point and each of the depth planes, and determine each of the second distances after the normalization as an estimated depth value of the pixel point relative to each of the depth planes;

[0144] For each of the depth planes, a normalization process is performed on the second distance between each of the pixel points and the depth plane, and a depth value weight of the depth plane is determined according to each of the second distances after the normalization process.

[0145] Optionally, the result determination unit 53 is further specifically configured to:

[0146] multiplying each of the second distances after normalization by the corresponding second distances before normalization;

[0147] Each multiplication result is linearly mapped to a depth value weight of the depth plane.

[0148] Optionally, the result determination unit 53 is further specifically configured to:

[0149] For each of the pixel points, a weighted sum is performed on the estimated depth value of the pixel point relative to each of the depth planes according to the depth value weight of each of the depth planes to obtain the depth value of the pixel point.

[0150] Optionally, a model training unit is further included, for:

[0151] Obtain sample images and reference images for model training, and predict sample depth maps corresponding to the sample images through a pre-built neural network structure;

[0152] reconstructing the sample image according to camera motion data between the sample image and the reference image, the reference image and the sample depth map;

[0153] Based on the sample image, the reconstructed sample image and the sample depth map, the neural network structure is trained to obtain the depth estimation model.

[0154] Optionally, the model training unit is specifically used for:

[0155] By means of the neural network structure, pixel features of each sample pixel in the sample image and plane features of each sample depth plane in the sample image are extracted; the sample depth plane is used to represent a plane in a viewing space of the sample image; and there is a third distance between the plane in the viewing space of the sample image and a camera that shoots the sample image;

[0156] The neural network structure is used to predict a sample depth map corresponding to the sample image according to the pixel feature of each sample pixel and the plane feature of each sample depth plane.

[0157] Optionally, the model training unit is further specifically used for:

[0158] Determine a fourth distance between each of the sample pixel points and each of the sample depth planes according to a pixel feature of each of the sample pixel points and a plane feature of each of the sample depth planes;

[0159] According to a fourth distance between each of the sample pixel points and each of the sample depth planes, a depth value of each of the sample pixel points is determined to obtain the sample depth map.

[0160] Optionally, the model training unit is further specifically used for:

[0161] Dot product processing is performed on the pixel feature of each of the sample pixel points and the plane feature of each of the sample depth planes to obtain a fourth distance between each of the sample pixel points and each of the sample depth planes.

[0162] Optionally, the model training unit is further specifically used for:

[0163] Determining, according to a fourth distance between each of the sample pixel points and each of the sample depth planes, an estimated depth value of each of the sample pixel points relative to each of the sample depth planes and a depth value weight of each of the sample depth planes;

[0164] The depth value of each of the sample pixel points is determined according to the estimated depth value of each of the sample pixel points relative to each of the sample depth planes and the depth value weight of each of the sample depth planes.

[0165] Optionally, the model training unit is further specifically used for:

[0166] For each of the sample pixel points, normalize the fourth distance between the sample pixel point and each of the sample depth planes, and determine each of the fourth distances after the normalization as an estimated depth value of the sample pixel point relative to each of the sample depth planes;

[0167] For each of the sample depth planes, a fourth distance between each of the sample pixel points and the sample depth plane is normalized, and a depth value weight of the sample depth plane is determined according to each of the fourth distances after the normalization.

[0168] Optionally, the model training unit is further specifically used for:

[0169] multiplying each of the fourth distances after normalization processing by the corresponding fourth distances before normalization processing;

[0170] Each multiplication result is linearly mapped to a depth value weight of the sample depth plane.

[0171] Optionally, the model training unit is further specifically used for:

[0172] For each of the sample pixel points, according to the depth value weight of each of the sample depth planes, a weighted sum is performed on the estimated depth value of the sample pixel point relative to each of the sample depth planes to obtain the depth value of the sample pixel point.

[0173] The image depth estimation device in the embodiment of the present disclosure can implement each process of the above-mentioned image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0174] An embodiment of the present disclosure further provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 601 and memory 602, and one or more applications or data may be stored in the memory 602. Among them, the memory 602 may be a short-term storage or a persistent storage. The application stored in the memory 602 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 601 may be configured to communicate with the memory 602 to execute a series of computer executable instructions in the memory 602 on the electronic device. The electronic device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input or output interfaces 605, one or more keyboards 606, etc.

[0175] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0176] Acquire an image for which depth estimation is to be performed, and extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; and there is a first distance between the plane in the viewing space of the image and a camera that captures the image;

[0177] Determine, by the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes;

[0178] The depth estimation model is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane, so as to obtain a depth map corresponding to the image.

[0179] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0180] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0181] Acquire an image for which depth estimation is to be performed, and extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; and there is a first distance between the plane in the viewing space of the image and a camera that captures the image;

[0182] Determine, by the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes;

[0183] The depth estimation model is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane, so as to obtain a depth map corresponding to the image.

[0184] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0185] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0186] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0187] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0188] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0189] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0190] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0191] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0192] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0194] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0195] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0196] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A method for estimating image depth, characterized in that: include: Acquire an image for which depth estimation is to be performed, and extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; and there is a first distance between the plane in the viewing space of the image and a camera that captures the image; Determine, by the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes; The depth estimation model is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane, so as to obtain a depth map corresponding to the image.

2. The method according to claim 1, characterized in that The extracting pixel features of each pixel in the image and plane features of each depth plane in the image includes: Performing feature extraction on the image to obtain pixel features of each pixel in the image; The plane features of each depth plane in the image are extracted according to the pixel features of each pixel.

3. The method according to claim 1, characterized in that The determining, according to the pixel feature of each pixel and the plane feature of each depth plane, the second distance between each pixel and each depth plane comprises: Dot product processing is performed on the pixel feature of each pixel and the plane feature of each depth plane to obtain a second distance between each pixel and each depth plane.

4. The method according to claim 1, characterized in that: The determining the depth value of each pixel point according to the second distance between each pixel point and each depth plane includes: Determining, according to a second distance between each pixel and each depth plane, an estimated depth value of each pixel relative to each depth plane and a depth value weight of each depth plane; The depth value of each pixel is determined according to the estimated depth value of each pixel relative to each depth plane and the depth value weight of each depth plane.

5. The method according to claim 4, characterized in that The step of determining, according to the second distance between each pixel point and each depth plane, an estimated depth value of each pixel point relative to each depth plane and a depth value weight of each depth plane comprises: For each of the pixel points, normalize the second distance between the pixel point and each of the depth planes, and determine each of the second distances after the normalization as an estimated depth value of the pixel point relative to each of the depth planes; For each of the depth planes, a normalization process is performed on the second distance between each of the pixel points and the depth plane, and a depth value weight of the depth plane is determined according to each of the second distances after the normalization process.

6. The method according to claim 5, characterized in that The step of determining the depth value weight of the depth plane according to each of the second distances after normalization includes: multiplying each of the second distances after normalization by the corresponding second distances before normalization; Each multiplication result is linearly mapped to a depth value weight of the depth plane.

7. The method according to claim 4, characterized in that Determining the depth value of each pixel point according to the estimated depth value of each pixel point relative to each depth plane and the depth value weight of each depth plane includes: For each of the pixel points, a weighted sum is performed on the estimated depth value of the pixel point relative to each of the depth planes according to the depth value weight of each of the depth planes to obtain the depth value of the pixel point.

8. The method according to claim 1, characterized in that: Also includes: Obtain sample images and reference images for model training, and predict sample depth maps corresponding to the sample images through a pre-built neural network structure; reconstructing the sample image according to camera motion data between the sample image and the reference image, the reference image and the sample depth map; Based on the sample image, the reconstructed sample image and the sample depth map, the neural network structure is trained to obtain the depth estimation model.

9. The method according to claim 8, characterized in that The predicting of the sample depth map corresponding to the sample image by using the pre-built neural network structure includes: By means of the neural network structure, pixel features of each sample pixel in the sample image and plane features of each sample depth plane in the sample image are extracted; the sample depth plane is used to represent a plane in a viewing space of the sample image; and there is a third distance between the plane in the viewing space of the sample image and a camera that shoots the sample image; The neural network structure is used to predict a sample depth map corresponding to the sample image according to the pixel feature of each sample pixel and the plane feature of each sample depth plane.

10. The method according to claim 9, characterized in that The predicting the sample depth map corresponding to the sample image according to the pixel feature of each sample pixel and the plane feature of each sample depth plane includes: Determine a fourth distance between each of the sample pixel points and each of the sample depth planes according to a pixel feature of each of the sample pixel points and a plane feature of each of the sample depth planes; According to a fourth distance between each of the sample pixel points and each of the sample depth planes, a depth value of each of the sample pixel points is determined to obtain the sample depth map.

11. The method according to claim 10, characterized in that The determining, according to the pixel feature of each sample pixel and the plane feature of each sample depth plane, a fourth distance between each sample pixel and each sample depth plane comprises: Dot product processing is performed on the pixel feature of each of the sample pixel points and the plane feature of each of the sample depth planes to obtain a fourth distance between each of the sample pixel points and each of the sample depth planes.

12. The method according to claim 10, characterized in that The determining the depth value of each of the sample pixel points according to the fourth distance between each of the sample pixel points and each of the sample depth planes includes: Determining, according to a fourth distance between each of the sample pixel points and each of the sample depth planes, an estimated depth value of each of the sample pixel points relative to each of the sample depth planes and a depth value weight of each of the sample depth planes; The depth value of each of the sample pixel points is determined according to the estimated depth value of each of the sample pixel points relative to each of the sample depth planes and the depth value weight of each of the sample depth planes.

13. The method according to claim 12, characterized in that The step of determining, according to a fourth distance between each of the sample pixel points and each of the sample depth planes, an estimated depth value of each of the sample pixel points relative to each of the sample depth planes and a depth value weight of each of the sample depth planes comprises: For each of the sample pixel points, normalize the fourth distance between the sample pixel point and each of the sample depth planes, and determine each of the fourth distances after the normalization as an estimated depth value of the sample pixel point relative to each of the sample depth planes; For each of the sample depth planes, a fourth distance between each of the sample pixel points and the sample depth plane is normalized, and a depth value weight of the sample depth plane is determined according to each of the fourth distances after the normalization.

14. The method according to claim 13, characterized in that The step of determining the depth value weight of the sample depth plane according to each of the fourth distances after normalization includes: multiplying each of the fourth distances after normalization processing by the corresponding fourth distances before normalization processing; Each multiplication result is linearly mapped to a depth value weight of the sample depth plane.

15. The method according to claim 12, characterized in that Determining the depth value of each of the sample pixel points according to the estimated depth value of each of the sample pixel points relative to each of the sample depth planes and the depth value weight of each of the sample depth planes comprises: For each of the sample pixel points, according to the depth value weight of each of the sample depth planes, a weighted sum is performed on the estimated depth value of the sample pixel point relative to each of the sample depth planes to obtain the depth value of the sample pixel point.

16. An image depth estimation device, characterized in that: include: A feature extraction unit, used to obtain an image to be depth estimated, and extract pixel features of each pixel in the image and plane features of each depth plane in the image through a pre-trained depth estimation model; the depth plane is used to represent a plane in a viewing space of the image; the plane in the viewing space of the image has a first distance from a camera that captures the image; a distance determination unit, configured to determine, by using the depth estimation model, a second distance between each of the pixel points and each of the depth planes according to a pixel feature of each of the pixel points and a plane feature of each of the depth planes; The result determination unit is used to determine the depth value of each pixel point according to the second distance between each pixel point and each depth plane through the depth estimation model, so as to obtain a depth map corresponding to the image.

17. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 15.

Images