Image depth estimation method, apparatus, device, and storage medium

Abstract

The embodiment of the present specification provides a kind of image depth estimation method, device, equipment and storage medium, wherein method includes: by depth estimation model, the depth information of target image is estimated, obtains the initial depth image corresponding to the target image;The training data of the depth estimation model includes sample image and the planarization depth image corresponding to the sample image;The point cloud data of the first object in the planarization depth image is pre-planarized processing;Locate problem pixel in the initial depth image;The pixel value of the problem pixel point is adjusted;Based on the initial depth image after adjustment, determine the final depth image corresponding to the target image. By the present embodiment, the accuracy of the depth estimation result of depth estimation model can be improved.

Description

Technical Field

[0001] This document relates to the field of deep learning technology, and in particular to an image depth estimation method, apparatus, device, and storage medium. Background Technology

[0002] Depth estimation of images yields depth information, which can be applied to various scenarios, such as 3D modeling. Currently, depth estimation is primarily performed using neural network-based models. However, the accuracy of these current models remains relatively low. Summary of the Invention

[0003] This specification provides an image depth estimation method, apparatus, device, and storage medium to improve the accuracy of depth estimation results from depth estimation models.

[0004] Firstly, embodiments of this specification provide an image depth estimation method, including:

[0005] The depth information of the target image is estimated using a depth estimation model to obtain an initial depth image corresponding to the target image; the training data of the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized.

[0006] Locate the problem pixel in the initial depth image;

[0007] Adjust the pixel values ​​of the problematic pixels;

[0008] Based on the adjusted initial depth image, the final depth image corresponding to the target image is determined.

[0009] Secondly, embodiments of this specification provide an image depth estimation apparatus, including:

[0010] The depth estimation unit is used to estimate the depth information of the target image through a depth estimation model to obtain an initial depth image corresponding to the target image; the training data of the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized.

[0011] A pixel localization unit is used to locate problem pixels in the initial depth image;

[0012] A pixel value adjustment unit is used to adjust the pixel value of the problematic pixel.

[0013] The result determination unit is used to determine the final depth image corresponding to the target image based on the adjusted initial depth image.

[0014] Thirdly, embodiments of this specification provide an electronic device, including:

[0015] Processor; and,

[0016] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in the first aspect above.

[0017] Fourthly, embodiments of this specification provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0018] Through one or more embodiments of this specification, firstly, the depth information of the target image is estimated using a depth estimation model to obtain an initial depth image corresponding to the target image. The training data of the depth estimation model includes sample images and corresponding planarized depth images. The point cloud data of the first object in the planarized depth image has been pre-planarized. Then, problem pixels are located in the initial depth image, and their pixel values ​​are adjusted. Finally, based on the adjusted initial depth image, the final depth image corresponding to the target image is determined. Therefore, in one or more embodiments of this specification, on the one hand, because the training data of the depth estimation model includes sample images and corresponding planarized depth images, and the point cloud data of the first object in the planarized depth image has been pre-planarized, the accuracy of the model training data is improved during the model training phase, thereby improving the accuracy of the depth estimation results of the trained depth estimation model. On the other hand, after estimating the depth information of the target image using the depth estimation model to obtain the initial depth image, the pixel values ​​of the problem pixels in the initial depth image are adjusted. Based on the adjusted initial depth image, the final depth image corresponding to the target image is obtained. Thus, by optimizing the estimation results of the depth estimation model, the accuracy of image depth estimation is further improved. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart illustrating an image depth estimation method provided in one embodiment of this specification;

[0021] Figure 2 A schematic diagram of a second object provided in one embodiment of this specification;

[0022] Figure 3 This is a schematic diagram illustrating the process of acquiring a planar depth image according to an embodiment of this specification;

[0023] Figure 4 A schematic flowchart of a depth estimation model provided in one embodiment of this specification;

[0024] Figure 5 This is a schematic diagram of the structure of a depth estimation model provided in one embodiment of this specification;

[0025] Figure 6 This is a schematic diagram of the structure of an image depth estimation device provided in one embodiment of this specification;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0028] This specification provides one or more embodiments of an image depth estimation method, apparatus, device, and storage medium to improve the accuracy of depth estimation results from depth estimation models. The image depth estimation method in this embodiment can be applied to the field of 3D modeling. To facilitate understanding of the various embodiments in this specification, a brief description of the 3D modeling process based on depth images is provided here. Taking indoor modeling as an example, multiple panoramic images can be captured from different viewpoints within the room, and a depth image corresponding to each panoramic image can be generated. Then, point cloud data corresponding to each depth image is obtained, and the point cloud data corresponding to each depth image is stitched together to establish a 3D model of the indoor scene.

[0029] Figure 1 This is a flowchart illustrating an image depth estimation method provided in one embodiment of this specification, as shown below. Figure 1As shown, the process includes:

[0030] Step S102: The depth information of the target image is estimated by the depth estimation model to obtain the initial depth image corresponding to the target image; the training data of the depth estimation model includes sample images and the planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized.

[0031] Step S104: Locate the problem pixel in the initial depth image;

[0032] Step S106: Adjust the pixel value of the problematic pixel;

[0033] Step S108: Based on the adjusted initial depth image, determine the final depth image corresponding to the target image.

[0034] In this embodiment, firstly, the depth information of the target image is estimated using a depth estimation model to obtain an initial depth image corresponding to the target image. The training data for the depth estimation model includes sample images and their corresponding planarized depth images. The point cloud data of the first object in the planarized depth image has been pre-planarized. Then, problem pixels are located in the initial depth image, and their pixel values ​​are adjusted. Finally, based on the adjusted initial depth image, the final depth image corresponding to the target image is determined. Therefore, in this embodiment, on the one hand, because the training data for the depth estimation model includes sample images and their corresponding planarized depth images, and the point cloud data of the first object in the planarized depth image has been pre-planarized, the accuracy of the model training data is improved during the model training phase, thereby improving the accuracy of the depth estimation result obtained by the trained depth estimation model. On the other hand, because after estimating the depth information of the target image using the depth estimation model to obtain the initial depth image, the pixel values ​​of the problem pixels in the initial depth image are also adjusted, and the final depth image corresponding to the target image is obtained based on the adjusted initial depth image, the accuracy of image depth estimation is further improved by optimizing the estimation result of the depth estimation model.

[0035] The image depth estimation method in this embodiment can be applied on the server side and executed by the server.

[0036] In step S102 above, a target image is acquired, and its depth information is estimated using a depth estimation model. The depth image output by the depth estimation model is then used as the initial depth image corresponding to the target image. In the scenario of 3D modeling a house, the target image can be an RGB panoramic image of the house captured from the shooting viewpoint. The target image is input into a pre-trained depth estimation model, which processes the target image and outputs the initial depth image corresponding to it. The initial depth image has the same size as the target image, and the pixels in the initial depth image correspond one-to-one with the pixels in the target image. The pixel value of each pixel in the initial depth image is the depth value of the corresponding pixel in the target image.

[0037] In this embodiment, the training data for the depth estimation model includes sample images and corresponding planarized depth images. The point cloud data of the first object in the planarized depth image has been pre-processed for planarization. The first object refers to the object whose point cloud data needs to be optimized. The first object can be considered as a large-area planar object, including but not limited to at least one of ceiling, floor, wall, door, window, archway, and doorway. By performing planarization processing on the point cloud data of the first object, the point cloud data of the first object can be more concentrated in the same plane, thereby improving the accuracy of depth estimation by the depth estimation model.

[0038] In steps S104 to S106 above, problematic pixels are located in the initial depth image, and their pixel values ​​are adjusted. The depth represented by the pixel values ​​of problematic pixels located in the initial depth image is not accurate enough. When using the pixel values ​​of problematic pixels to generate point cloud data and model in a 3D modeling scenario, this may reduce the model's accuracy. Therefore, here, problematic pixels are located in the initial depth image, and their pixel values ​​are adjusted to correct the depth represented by the pixel values ​​of the problematic pixels.

[0039] In one embodiment, the number of target images is the same as the number of initial depth images, and the two correspond one-to-one. Taking any initial depth image as an example, locating the problem pixel in the initial depth image specifically includes:

[0040] Locate the second object in the initial depth image; the pixel of the second object in the initial depth image is the problem pixel.

[0041] and / or

[0042] In the initial depth image, identify pixels with pixel values ​​greater than a preset pixel threshold; these identified pixels are considered problem pixels.

[0043] In this embodiment, any one or two of the methods can be used to locate problem pixels in the initial depth image. In one embodiment, pixels with pixel values ​​greater than a preset pixel threshold can be identified in the initial depth image first, and these identified pixels can be used as a portion of problem pixels. Then, a second object can be located in the initial depth image, and the pixels of the second object in the initial depth image can be used as another portion of problem pixels. These two portions of problem pixels can be used together as all the problem pixels.

[0044] In one embodiment, taking an indoor modeling scene as an example, the second object can be a door in the image and the image content contained within the door, or the second object can be a window in the image and the image content contained within the window. Since the accuracy of the predicted depth value of an object in the image is poor when photographing an object in another area through a door, window, or opening because the object is far from the shooting location, the second object is located in the initial depth image, and the pixels of the second object in the initial depth image are used as problem pixels.

[0045] Figure 2 A schematic diagram of a second object provided in one embodiment of this specification, as shown below. Figure 2 As shown, the second object can be the door in the image and the image content contained within the door.

[0046] When the subject is far from the shooting location, the accuracy of the depth value predicted in the image is also poor. Based on this, in one embodiment, pixels with pixel values ​​greater than a preset pixel threshold in the initial depth image are regarded as problem pixels. The depth represented by the pixel value of the problem pixels is too large and may not be accurate enough.

[0047] This embodiment enables accurate and rapid location of problem pixels in the initial depth image, improving the processing efficiency of the initial depth image.

[0048] In one embodiment, taking the location of a second object in an initial depth image as an example, locating the second object in the initial depth image can be as follows:

[0049] Obtain the position information of the second object in the target image;

[0050] Based on the position information of the second object in the target image, determine the position information of the second object in the initial depth image;

[0051] Based on the position information of the second object in the initial depth image, locate the second object in the initial depth image.

[0052] Since the second object is located in the initial depth image, it is also located in the target image. First, the position information of the second object in the target image is determined, which can be represented by the pixel coordinates of the second object. Because the target image and the initial depth image are pixel-to-pixel correspondences, the position information of the second object in the target image is equal to its position information in the initial depth image. Therefore, based on the position information of the second object in the target image, the second object can be located in the initial depth image.

[0053] In one embodiment, the positional information of the second object can be obtained in each target image to facilitate the localization of the second object in each initial depth image. Taking any target image as an example, the process of obtaining the positional information of the second object in the target image specifically includes:

[0054] The target image is processed by a pre-trained semantic segmentation model to obtain semantic segmentation result data corresponding to the target image; the semantic segmentation result data indicates the semantic information of the object represented by each pixel in the target image;

[0055] Based on the semantic segmentation results data corresponding to the target image, the location information of the second object in the target image is determined.

[0056] A semantic segmentation model is pre-trained, and the training data for this model is labeled with a second object to enable it to recognize that object. The target image is then input into the semantic segmentation model, which processes the image and outputs semantic segmentation results. These results indicate the semantic information of the object represented by each pixel in the target image.

[0057] The semantic segmentation result data corresponding to the target image can be a semantic segmentation image in which the pixels of the target image have a one-to-one correspondence. The pixel value of each pixel in the image is used to represent the object category to which the pixel belongs. For example, if the pixel value is 2, it means that the pixel is the second object. If the pixel value is other values, it means that the pixel is the other object.

[0058] Based on this, all pixels contained in the second object can be located in the semantic segmentation image. According to the one-to-one correspondence between the pixels in the semantic segmentation image and the pixels in the target image, all pixels contained in the second object can be located in the target image. Based on the pixel coordinates of all pixels contained in the second object in the target image, the position information of the second object in the target image can be determined. This position information can be represented in the form of pixel coordinates.

[0059] As can be seen, in this embodiment, by using semantic segmentation, the second object can be located in the target image, and then the second object can be located in the initial depth image. Object localization based on the semantic segmentation model has the advantages of high localization accuracy and high localization speed and efficiency.

[0060] In step S106 above, the pixel values ​​of the problematic pixels are also adjusted. Specifically, adjusting the pixel values ​​of the problematic pixels can be done by modifying them to preset pixel values. For example, the pixel values ​​of the problematic pixels can be modified to 0 or any negative number so that the problematic pixels do not have effective depth information. This allows the depth of the problematic pixels to be removed from the initial depth image, ensuring that the point cloud data corresponding to the problematic pixels does not participate in the 3D modeling process, thereby improving the accuracy of the model.

[0061] In step S108 above, the final depth image corresponding to the target image is obtained based on the adjusted initial depth image. For example, the adjusted initial depth image is used as the final depth image corresponding to the target image, that is, as the depth estimation result of the target image.

[0062] As can be seen, in this embodiment, on the one hand, since the training data of the depth estimation model includes sample images and corresponding planarized depth images, and the point cloud data of the first object in the planarized depth image has been pre-planarized, the accuracy of the model training data is improved during the model training stage, thereby improving the accuracy of the depth estimation result of the trained depth estimation model. On the other hand, since the initial depth image is obtained by estimating the depth information of the target image through the depth estimation model, the pixel values ​​of the problem pixels in the initial depth image are adjusted. Based on the adjusted initial depth image, the final depth image corresponding to the target image is obtained. Thus, by optimizing the estimation result of the depth estimation model, the accuracy of image depth estimation is further improved.

[0063] In the field of 3D modeling, applying the above image depth estimation methods to estimate the depth information of images and converting the depth information into point cloud data for modeling can effectively improve the accuracy of the established 3D model.

[0064] The following details the process of acquiring the training data for the depth estimation model described above. The training data for the depth estimation model includes sample images and corresponding planarized depth images. The point cloud data of the first object in the planarized depth image has been pre-planarized.

[0065] Figure 3 This is a schematic diagram illustrating the process of acquiring a planar depth image according to an embodiment of this specification, as shown below. Figure 3 As shown, the process includes:

[0066] Step S302: Obtain the sample image and the structured light depth image corresponding to the sample image generated by the structured light device;

[0067] Step S304: Planarize the point cloud data of the first object in the structured light depth image;

[0068] Step S306: Map the point cloud data of the first object after planarization back to the structured light depth image, and use the mapped structured light depth image as the planarized depth image.

[0069] As can be seen, this embodiment optimizes the model training data by planarizing the point cloud data of the first object in the structured light depth image, and then trains the depth estimation model based on the optimized model training data, effectively improving the accuracy of the depth estimation results of the depth estimation model.

[0070] In step S302 above, a sample image and a structured light depth image corresponding to the sample image generated by the structured light device are acquired. Taking the 3D modeling of the interior environment of a house as an example, the sample image can be an RGB panoramic image of the interior of the house taken from different shooting viewpoints within the house, and the structured light depth image can be a depth image taken from each shooting viewpoint using a specific device such as a structured light device. One shooting viewpoint can capture one sample image and one structured light depth image. The pixels in the sample image correspond one-to-one with the pixels in the corresponding structured light depth image. Furthermore, the sample image and the corresponding structured light depth image have the same image size and capture the same content. The difference lies in that the pixel values ​​in the sample image represent the RGB data of the captured content, while the pixel values ​​in the structured light depth image represent the depth data of the captured content. In this step, multiple sample images and multiple structured light depth images can be acquired.

[0071] In step S304 above, the point cloud data of the first object in the structured light depth image is subjected to planarization processing. In one embodiment, the planarization processing of the point cloud data of the first object in the structured light depth image specifically involves: locating the first object in the structured light depth image, obtaining the point cloud data of the first object, and performing planarization processing on the point cloud data of the first object. It is understood that the pixels in the sample image and the structured light depth image correspond one-to-one, and both the sample image and the structured light depth image contain the first object.

[0072] In one embodiment, the first object can be located in each structured light depth image. Taking any structured light depth image as an example, the process of locating the first object in the structured light depth image specifically includes:

[0073] Obtain the position information of the first object in the sample image;

[0074] Based on the position information of the first object in the sample image, determine the position information of the first object in the structured light depth image;

[0075] Based on the position information of the first object in the structured light depth image, the first object is located in the structured light depth image.

[0076] Since the sample image contains the first object, its position information within the sample image is first determined. This position information can be represented by the pixel coordinates of the first object. Because the sample image and the structured light depth image are images with a one-to-one correspondence between pixels, the position information of the first object in the sample image is equal to its position information in the structured light depth image. Therefore, based on the position information of the first object in the sample image, the first object can be located in the structured light depth image.

[0077] In one embodiment, the positional information of the first object can be obtained in each sample image to facilitate the localization of the first object in each structured light depth image. Taking any sample image as an example, the process of obtaining the positional information of the first object in the sample image specifically includes:

[0078] The sample image is processed by a pre-trained semantic segmentation model to obtain the semantic segmentation result data corresponding to the sample image; the semantic segmentation result data indicates the semantic information of the object represented by each pixel in the sample image;

[0079] Based on the semantic segmentation results data corresponding to the sample image, determine the location information of the first object in the sample image.

[0080] A pre-trained semantic segmentation model is used. During training, the first object can be labeled in the training data to enable the semantic segmentation model to recognize the first object in an image. First, the sample image is input into the pre-trained semantic segmentation model. The model processes the sample image to obtain the corresponding semantic segmentation result data. The semantic segmentation result data can be a semantic segmentation image where pixels correspond one-to-one with pixels in the sample image. The pixel value of each pixel in this image represents the object category to which that pixel belongs. For example, a pixel value of 1 indicates that the pixel belongs to the first object, while other values ​​indicate that the pixel belongs to other objects.

[0081] Based on this, all pixels contained in the first object can be located in the semantic segmentation image. According to the correspondence between the pixels in the semantic segmentation image and the pixels in the sample image, all pixels contained in the first object can be located in the sample image. According to the pixel coordinates of all pixels contained in the first object in the sample image, the position information of the first object in the sample image can be determined. This position information can be represented in the form of pixel coordinates.

[0082] As can be seen, in this embodiment, the first object can be located in the sample image through the above process, and then the first object can be located in the structured light depth image. The object location based on the semantic segmentation model has the advantages of high positioning accuracy and fast positioning speed and efficiency.

[0083] In the above process, point cloud data of the first object is also acquired. After locating the first object in the depth image, the point cloud data of the first object is acquired based on the principle of converting the depth map into point cloud data.

[0084] As described above, the first object can be considered a large-area planar object. By optimizing the point cloud data of the first object, the point cloud data can be more concentrated on the same plane, making the plane of the first object in the established 3D model smoother. Based on this, the point cloud data of the first object is also planarized in the above process, and the structured light depth image is adjusted based on the planarized point cloud data of the first object.

[0085] It is understood that the point cloud data of the first object in each structured light depth image is subjected to planarization processing. Based on this, taking the point cloud data of the first object in any structured light depth image as an example, the above-mentioned planarization process of the point cloud data of the first object specifically includes:

[0086] Noise data is identified in the point cloud data of the first object, and the location of the noise data is outside the target plane corresponding to the first object;

[0087] Remove noise data from the point cloud data of the first object, or adjust the noise data so that the position corresponding to the noise data is on the target plane corresponding to the first object.

[0088] The first object, being a large-area planar object, has a corresponding target plane. For example, if the first object is a ceiling, the plane containing the ceiling is the target plane corresponding to the first object. Due to errors in the point cloud data of the first object, a small number of point cloud data points may be located outside the target plane corresponding to the first object. Therefore, this section identifies the point cloud data points located outside the target plane corresponding to the first object in the point cloud data of the first object and treats these identified point cloud data as noise data.

[0089] Next, noise data is removed from the point cloud data of the first object. By removing noise data, the positions represented by the remaining point cloud data of the first object are all located on the target plane corresponding to the first object, making the plane of the first object in the established 3D model smoother.

[0090] Alternatively, the noise data can be adjusted to bring the positions represented by the noise data back to the target plane corresponding to the first object. By adjusting the data values ​​of the noise data, such as adjusting the XYZ coordinates of the noise data, the positions represented by the noise data can be brought back to the target plane corresponding to the first object. This makes the plane of the first object in the established 3D model smoother while retaining as much point cloud data as possible.

[0091] In one embodiment, taking the point cloud data of a first object in any structured light depth image as an example, noise data is identified in the point cloud data of the first object, specifically as follows:

[0092] Normal vector estimation is performed on the point cloud data of the first object to obtain the normal vector of each point cloud data;

[0093] Determine the normal vector of the target plane based on the normal vector of each point cloud data.

[0094] Based on the normal vector of each point cloud data and the normal vector of the target plane, noise data is determined. Noise data refers to point cloud data whose normal vectors do not match the normal vectors of the target plane.

[0095] First, a pre-defined normal vector estimation algorithm is used to estimate the normal vector for each point cloud data point of the first object, resulting in the normal vector for each point cloud data point. Then, based on the normal vector of each point cloud data point, the normal vector of the target plane is determined. For example, if the normal vectors corresponding to most point cloud data points are the same, this same normal vector is taken as the normal vector of the target plane. Finally, among all the point cloud data points, those whose normal vectors are different from the normal vector of the target plane are identified. Since the normal vectors of these point cloud data points are not the normal vectors of the target plane, the positions represented by these point cloud data points must be located outside the target plane corresponding to the first object; therefore, these point cloud data points are considered noise data.

[0096] In this embodiment, noise data can be quickly identified in the point cloud data of the first object by normal vector estimation, thereby improving the efficiency of planarization processing of the point cloud data of the first object.

[0097] Figure 3In step S306, the point cloud data of the first object after planarization is mapped back to the structured light depth image, and the mapped structured light depth image is used as the planarized depth image. Specifically, this can be achieved by converting the point cloud data into a depth map, mapping the point cloud data of the first object after planarization onto the structured light depth image. Since the point cloud data of the first object has changed, the depth value of the first object on the structured light depth image will also change accordingly, thus using the mapped structured light depth image as the planarized depth image.

[0098] After obtaining the planarized depth images corresponding to the sample images and sample graphics through the above process, the sample images and planarized depth images can be used as training data for the depth estimation model in a one-to-one correspondence. Alternatively, a subset of sample images and planarized depth images can be selected from the sample images and planarized depth images and used as training data for the depth estimation model in a one-to-one correspondence.

[0099] As can be seen, this embodiment optimizes the structured light depth image by performing planarization processing on the point cloud data of the first object in the structured light depth image. The optimized structured light depth image is then used as training data for the depth estimation model, which effectively improves the accuracy of the training data for the depth estimation model and thus improves the accuracy of the depth estimation results.

[0100] The training process of the depth estimation model described above is explained in detail below. Figure 4 This is a flowchart illustrating a depth estimation model provided in one embodiment of this specification, as shown below. Figure 4 As shown, the process includes:

[0101] Step S402: Obtain the main training data for the depth estimation model. The main training data includes the sample images and planarized depth images mentioned above.

[0102] Step S404: Obtain auxiliary training data for the depth estimation model. The auxiliary training data includes at least one of the semantic segmentation result data and normal vector estimation result data corresponding to the sample images.

[0103] Step S406: Train a depth estimation model based on the main training data and auxiliary training data; the depth estimation model is used to estimate the depth of the image, and is also used to perform semantic segmentation and / or normal vector estimation of the image.

[0104] In this embodiment, the main training data for the depth estimation model includes the aforementioned sample images and the aforementioned planarized depth images. The auxiliary training data for the depth estimation model includes at least one of the semantic segmentation result data and normal vector estimation result data corresponding to the sample images. The depth estimation model is trained based on the main training data and the auxiliary training data, enabling it to perform semantic segmentation and / or normal vector estimation in addition to image depth estimation. Through a multi-task parallel depth estimation model architecture, the computational power of the depth estimation model can be improved, thereby auxiliaryly improving the performance of the depth estimation model in depth estimation and increasing the accuracy of the depth estimation results.

[0105] In step S402 above, sample images and planarized depth images are acquired as the main training data for the depth estimation model. Using the main training data, the depth estimation model's ability to estimate the depth of images can be trained.

[0106] In step S404 above, at least one of the semantic segmentation result data and normal vector estimation result data corresponding to the sample image is obtained as auxiliary training data for the depth estimation model. The semantic segmentation result data corresponding to the sample image can be achieved through a pre-trained semantic segmentation model. The semantic segmentation result data of the sample image can include a semantic segmentation image. The pixel values ​​in the semantic segmentation image are used to represent the semantic information of the object to which the corresponding pixel belongs. For example, a pixel value of 1 indicates that the corresponding pixel belongs to object 1, and a pixel value of 2 indicates that the corresponding pixel belongs to object 2. There is a correspondence between the pixels of the semantic segmentation image and the sample image. Therefore, based on the pixel values ​​in the semantic segmentation image and this correspondence, the object to which each pixel in the sample image belongs can be determined, achieving the effect of semantic segmentation. By labeling the training data of the semantic segmentation model, the ability to identify multiple objects in the sample image can be achieved.

[0107] The normal vector estimation result data corresponding to the sample image includes the normal vector corresponding to each pixel in the sample image. The normal vector of the sample image can be estimated using any normal vector estimation algorithm; no restrictions are imposed here.

[0108] In step S406 above, a depth estimation model is trained based on the main training data and auxiliary training data. When the auxiliary training data includes semantic segmentation result data corresponding to sample images, the depth estimation model is used to perform depth estimation and semantic segmentation of the images. When the auxiliary training data includes normal vector estimation result data corresponding to sample images, the depth estimation model is used to perform depth estimation, semantic segmentation, and normal vector estimation of the images. When the auxiliary training data includes semantic segmentation result data and normal vector estimation result data corresponding to sample images, the depth estimation model is used to perform depth estimation, semantic segmentation, and normal vector estimation of the images.

[0109] During the model training phase, auxiliary training data can be selected as needed to train the depth estimation model. For example, taking a scenario where auxiliary training data includes semantic segmentation results and normal vector estimation results corresponding to sample images, and the depth estimation model is used to perform depth estimation, semantic segmentation, and normal vector estimation on images, the specific process of training the depth estimation model based on the main training data and auxiliary training data includes:

[0110] The feature extraction module in the depth estimation model is used to process the sample image to obtain its image features.

[0111] The feature learning module in the depth estimation model learns the depth information, semantic segmentation information, and normal vectors of the sample images based on their image features.

[0112] By using a preset loss function, the learned depth information, semantic segmentation information, and normal vectors are compared with the planarized depth image, semantic segmentation result data, and normal vector estimation result data in sequence. The model parameters of the depth estimation model are adjusted based on the comparison results.

[0113] In one embodiment, the feature learning module includes a depth estimation module, a semantic segmentation module, and a normal vector estimation module. Accordingly, through the feature learning module in the depth estimation model, based on the image features of the sample image, the depth information, semantic segmentation information, and normal vector of the sample image are learned, specifically:

[0114] The depth estimation module in the depth estimation model learns the depth information of the sample images based on their image features.

[0115] The semantic segmentation module in the depth estimation model learns the semantic segmentation information of the sample images based on their image features.

[0116] The normal vector of a sample image is learned by using the normal vector estimation module in the depth estimation model, based on the image features of the sample image.

[0117] Figure 5This is a schematic diagram of the structure of a depth estimation model provided in one embodiment of this specification, as shown below. Figure 5 As shown, the depth estimation model includes a feature extraction module, a depth estimation module, a semantic segmentation module, and a normal vector estimation module. During the model training phase, sample images, planarized depth images, semantic segmentation results, and normal vector estimation results are input into the depth estimation model during training. The feature extraction module in the depth estimation model extracts features from the sample images through convolution operations to obtain the image features of the sample images. The image features of the sample images include texture features, geometric structure features, etc.

[0118] The feature extraction module transmits the extracted features to the depth estimation module, semantic segmentation module, and normal vector estimation module, respectively. The depth estimation module learns the depth information of the sample image through convolution operations based on its image features. This depth information can include the depth value of each pixel. The semantic segmentation module learns the semantic segmentation information of the sample image through convolution operations based on its image features. This semantic segmentation information can include the semantic category information of the object corresponding to each pixel. The normal vector estimation module learns the normal vector of the sample image through convolution operations based on its image features. This normal vector can include the value of the normal vector of each pixel.

[0119] A loss function is pre-defined for the depth estimation model. This loss function compares the learned depth information with the planar depth image in the main training data, compares the learned semantic segmentation information with the semantic segmentation results in the auxiliary training data, and compares the learned normal vector with the normal vector estimation results in the auxiliary training data. Based on these three comparison results, the model parameters of the depth estimation model are adjusted in reverse to minimize the loss, thereby obtaining a trained depth estimation model.

[0120] The above example, using the example of auxiliary training data including semantic segmentation results and normal vector estimation results corresponding to sample images, and a depth estimation model used for depth estimation, semantic segmentation, and normal vector estimation of images, illustrates the structure and training process of a depth estimation model. It is understandable that when auxiliary training data includes semantic segmentation results corresponding to sample images, and the depth estimation model is used for depth estimation and semantic segmentation of images, the corresponding model structure, compared to... Figure 3 The structure in the model will lack a normal vector estimation module, and the model training process will also lack the corresponding training process for the normal vector estimation module. Similarly, when the auxiliary training data includes the normal vector estimation results corresponding to the sample images, and the depth estimation model is used to perform depth and normal vector estimation on the image, the corresponding model structure will lack the normal vector estimation module compared to the previous model. Figure 3The structure in this case will lack a semantic segmentation module, and the model training process will also lack the corresponding process for training the semantic segmentation module. The model structure and training process in these two cases can be referred to the above introduction. Further details will not be provided here.

[0121] During the model application phase, the semantic segmentation module and the normal vector estimation module can be selectively retained from the trained depth estimation module according to requirements. For example, when only depth estimation is needed, only the depth estimation module can be retained in the trained depth estimation model; when both depth estimation and semantic segmentation are needed, both can be retained; when both depth estimation and normal vector estimation are needed, both can be retained; and when all three are needed, the depth estimation module, semantic segmentation module, and normal vector estimation module can be retained. The feature extraction module, as a fundamental module, must be retained in the depth estimation model in all cases.

[0122] Although the semantic segmentation module and normal vector estimation module can be selectively retained in the trained depth estimation module according to the needs during the model application stage, since at least one of the semantic segmentation module and normal vector estimation module is trained in addition to the depth estimation module and feature extraction module during the model training process, the computational power of the depth estimation model can be improved by using a multi-task parallel depth estimation model architecture and a multi-module parallel training method. This allows the feature extraction module in the depth estimation model to extract more features with higher accuracy and a greater number that meet various subsequent processing requirements, thereby helping to improve the performance of the depth estimation model in depth estimation and improving the accuracy of the depth estimation results.

[0123] The image depth estimation method has been described in detail above. Corresponding to the image depth estimation method described above, an embodiment of this specification also provides an image depth estimation apparatus for implementing the method. Figure 6 This is a schematic diagram of the structure of an image depth estimation device provided in one embodiment of this specification, as shown below. Figure 6 As shown, the device includes:

[0124] The depth estimation unit 61 is used to estimate the depth information of the target image through a depth estimation model to obtain an initial depth image corresponding to the target image; the training data of the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized.

[0125] Pixel localization unit 62 is used to locate problem pixels in the initial depth image;

[0126] The pixel value adjustment unit 63 is used to adjust the pixel value of the problematic pixel.

[0127] The result determination unit 64 is used to determine the final depth image corresponding to the target image based on the adjusted initial depth image.

[0128] Optionally, the pixel positioning unit 62 is specifically used for:

[0129] Locate a second object in the initial depth image, wherein the pixel of the second object in the initial depth image is the problem pixel; and / or

[0130] In the initial depth image, pixels with pixel values ​​greater than a preset pixel threshold are identified as problem pixels.

[0131] Optionally, the pixel positioning unit 62 is also specifically used for:

[0132] Obtain the position information of the second object in the target image;

[0133] Based on the position information of the second object in the target image, determine the position information of the second object in the initial depth image;

[0134] The second object is located in the initial depth image based on its position information in the initial depth image.

[0135] Optionally, the pixel positioning unit 62 is also specifically used for:

[0136] The target image is processed by a pre-trained semantic segmentation model to obtain semantic segmentation result data corresponding to the target image; the semantic segmentation result data indicates the semantic information of the object represented by each pixel in the target image;

[0137] Based on the semantic segmentation result data corresponding to the target image, the location information of the second object in the target image is determined.

[0138] Optionally, the pixel value adjustment unit 63 is specifically used for:

[0139] Modify the pixel value of the problematic pixel to a preset pixel value.

[0140] Optionally, it also includes an image generation unit for:

[0141] Acquire the sample image and the structured light depth image corresponding to the sample image generated by the structured light device;

[0142] The point cloud data of the first object in the structured light depth image is planarized.

[0143] The point cloud data of the first object after planarization is mapped back to the structured light depth image, and the mapped structured light depth image is used as the planarized depth image.

[0144] Optionally, the image generation unit is specifically used for:

[0145] Locate the first object in the structured light depth image;

[0146] Obtain the point cloud data of the first object;

[0147] The point cloud data of the first object is flattened.

[0148] Optionally, the image generation unit is also specifically used for:

[0149] Obtain the position information of the first object in the sample image;

[0150] Based on the position information of the first object in the sample image, determine the position information of the first object in the structured light depth image;

[0151] Based on the position information of the first object in the structured light depth image, the first object is located in the structured light depth image.

[0152] Optionally, the image generation unit is also specifically used for:

[0153] The sample image is processed by a pre-trained semantic segmentation model to obtain semantic segmentation result data corresponding to the sample image. The semantic segmentation result data indicates the semantic information of the object represented by each pixel in the sample image.

[0154] Based on the semantic segmentation result data corresponding to the sample image, the location information of the first object in the sample image is determined.

[0155] Optionally, the image generation unit is also specifically used for:

[0156] Noise data is identified in the point cloud data of the first object, wherein the location of the noise data is located outside the target plane corresponding to the first object;

[0157] The noise data can be deleted from the point cloud data of the first object, or the noise data can be adjusted so that the position corresponding to the noise data is on the target plane corresponding to the first object.

[0158] Optionally, the image generation unit is also specifically used for:

[0159] Normal vector estimation is performed on the point cloud data of the first object to obtain the normal vector of each point cloud data.

[0160] The normal vector of the target plane is determined based on the normal vector of each point cloud data.

[0161] Based on the normal vector of each point cloud data and the normal vector of the target plane, the noise data is determined, wherein the noise data is the point cloud data whose normal vector is inconsistent with the normal vector of the target plane.

[0162] Optionally, it also includes a model training unit for:

[0163] Obtain the master training data for the depth estimation model, wherein the master training data includes the sample images and the planarized depth images;

[0164] Obtain auxiliary training data for the depth estimation model, wherein the auxiliary training data includes at least one of the semantic segmentation result data and normal vector estimation result data corresponding to the sample image;

[0165] The depth estimation model is trained based on the main training data and the auxiliary training data; the depth estimation model is used to perform depth estimation on the image, and the depth estimation model is also used to perform semantic segmentation and / or normal vector estimation on the image.

[0166] Optionally, the auxiliary training data includes the semantic segmentation result data and the normal vector estimation result data; the depth estimation model is used to perform depth estimation, semantic segmentation, and normal vector estimation on the image; the model training unit is specifically used for:

[0167] The feature extraction module in the depth estimation model processes the sample image to obtain its image features.

[0168] The feature learning module in the depth estimation model learns the depth information, semantic segmentation information, and normal vector of the sample image based on the image features of the sample image.

[0169] By using a preset loss function, the learned depth information, semantic segmentation information, and normal vector are compared sequentially with the planarized depth image, the semantic segmentation result data, and the normal vector estimation result data. The model parameters of the depth estimation model are adjusted based on the comparison results.

[0170] Optionally, the feature learning module includes: a depth estimation module, a semantic segmentation module, and a normal vector estimation module; the model training unit is further specifically used for:

[0171] The depth estimation module in the depth estimation model learns the depth information of the sample image based on its image features.

[0172] The semantic segmentation module in the depth estimation model learns the semantic segmentation information of the sample image based on its image features.

[0173] The normal vector of the sample image is learned by the normal vector estimation module in the depth estimation model based on the image features of the sample image.

[0174] The image depth estimation device in this embodiment can implement the various processes of the aforementioned image depth estimation method embodiments and achieve the same effect and function, which will not be repeated here.

[0175] This specification also provides an electronic device according to one embodiment. Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification, such as... Figure 7 As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 701 and memories 702, with the memory 702 storing one or more application programs or data. The memory 702 can be temporary or persistent storage. The application programs stored in the memory 702 may include one or more modules (not shown), each module including a series of computer-executable instructions within the electronic device. Furthermore, the processor 701 may be configured to communicate with the memory 702, executing the series of computer-executable instructions stored in the memory 702 on the electronic device. The electronic device may also include one or more power supplies 703, one or more wired or wireless network interfaces 704, one or more input or output interfaces 705, one or more keyboards 706, etc.

[0176] In one specific embodiment, the electronic device is an image depth estimation device, specifically a backend server, including a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following process:

[0177] The depth information of the target image is estimated using a depth estimation model to obtain an initial depth image corresponding to the target image; the training data of the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized.

[0178] Locate the problem pixel in the initial depth image;

[0179] Adjust the pixel values ​​of the problematic pixels;

[0180] Based on the adjusted initial depth image, the final depth image corresponding to the target image is determined.

[0181] The image depth estimation device in the embodiments of this specification can implement the various processes of the aforementioned image depth estimation method embodiments and achieve the same effect and function, which will not be repeated here.

[0182] Another embodiment of this specification provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process:

[0183] The depth information of the target image is estimated using a depth estimation model to obtain an initial depth image corresponding to the target image; the training data of the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized.

[0184] Locate the problem pixel in the initial depth image;

[0185] Adjust the pixel values ​​of the problematic pixels;

[0186] Based on the adjusted initial depth image, the final depth image corresponding to the target image is determined.

[0187] The storage medium in the embodiments of this specification can implement the various processes of the aforementioned image depth estimation method embodiments and achieve the same effects and functions, which will not be repeated here.

[0188] In various embodiments of this specification, the computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0189] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0190] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0191] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0192] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0193] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0194] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0195] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0196] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0197] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0198] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0199] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0200] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. An image depth estimation method, characterized in that, include: The depth information of the target image is estimated using a depth estimation model to obtain an initial depth image corresponding to the target image. The training data for the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized. Locate the problem pixel in the initial depth image; Adjust the pixel values ​​of the problematic pixels; Based on the adjusted initial depth image, determine the final depth image corresponding to the target image; The planarized depth image corresponding to the sample image is obtained through the following steps: Acquire the sample image and the structured light depth image corresponding to the sample image generated by the structured light device; The point cloud data of the first object in the structured light depth image is planarized. The point cloud data of the first object after planarization is mapped back to the structured light depth image, and the mapped structured light depth image is used as the planarized depth image.

2. The method according to claim 1, characterized in that, Locating the problem pixel in the initial depth image includes: Locate a second object in the initial depth image, wherein the pixel of the second object in the initial depth image is the problem pixel; and / or In the initial depth image, pixels with pixel values ​​greater than a preset pixel threshold are identified as problem pixels.

3. The method according to claim 2, characterized in that, Locating the second object in the initial depth image includes: Obtain the position information of the second object in the target image; Based on the position information of the second object in the target image, determine the position information of the second object in the initial depth image; The second object is located in the initial depth image based on its position information in the initial depth image.

4. The method according to claim 3, characterized in that, The step of obtaining the position information of the second object in the target image includes: The target image is processed by a pre-trained semantic segmentation model to obtain semantic segmentation result data corresponding to the target image; the semantic segmentation result data indicates the semantic information of the object represented by each pixel in the target image; Based on the semantic segmentation result data corresponding to the target image, the location information of the second object in the target image is determined.

5. The method according to claim 1, characterized in that, The adjustment of the pixel value of the problematic pixel includes: Modify the pixel value of the problematic pixel to a preset pixel value.

6. The method according to claim 1, characterized in that, The planarization process for the point cloud data of the first object in the structured light depth image includes: Locate the first object in the structured light depth image; Obtain the point cloud data of the first object; The point cloud data of the first object is processed into a plane.

7. The method according to claim 6, characterized in that, Locating the first object in the structured light depth image includes: Obtain the position information of the first object in the sample image; Based on the position information of the first object in the sample image, determine the position information of the first object in the structured light depth image; Based on the position information of the first object in the structured light depth image, the first object is located in the structured light depth image.

8. The method according to claim 7, characterized in that, The step of obtaining the position information of the first object in the sample image includes: The sample image is processed by a pre-trained semantic segmentation model to obtain semantic segmentation result data corresponding to the sample image. The semantic segmentation result data indicates the semantic information of the object represented by each pixel in the sample image. Based on the semantic segmentation result data corresponding to the sample image, the location information of the first object in the sample image is determined.

9. The method according to claim 6, characterized in that, The planarization process for the point cloud data of the first object includes: Noise data is identified in the point cloud data of the first object, wherein the location of the noise data is located outside the target plane corresponding to the first object; The noise data can be deleted from the point cloud data of the first object, or the noise data can be adjusted so that the position corresponding to the noise data is on the target plane corresponding to the first object.

10. The method according to claim 9, characterized in that, The step of identifying noise data in the point cloud data of the first object includes: Normal vector estimation is performed on the point cloud data of the first object to obtain the normal vector of each point cloud data. The normal vector of the target plane is determined based on the normal vector of each point cloud data. Based on the normal vector of each point cloud data and the normal vector of the target plane, the noise data is determined, wherein the noise data is the point cloud data whose normal vector is inconsistent with the normal vector of the target plane.

11. The method according to claim 1, characterized in that, The depth estimation model is trained through the following steps: Obtain the master training data for the depth estimation model, wherein the master training data includes the sample images and the planarized depth images; Obtain auxiliary training data for the depth estimation model, wherein the auxiliary training data includes at least one of the semantic segmentation result data and normal vector estimation result data corresponding to the sample image; The depth estimation model is trained based on the main training data and the auxiliary training data; The depth estimation model is used to estimate the depth of an image, and it is also used for semantic segmentation and / or normal vector estimation of the image.

12. The method according to claim 11, characterized in that, The auxiliary training data includes the semantic segmentation result data and the normal vector estimation result data; the depth estimation model is used to perform depth estimation, semantic segmentation, and normal vector estimation on the image; The step of training the depth estimation model based on the main training data and the auxiliary training data includes: The feature extraction module in the depth estimation model processes the sample image to obtain its image features. The feature learning module in the depth estimation model learns the depth information, semantic segmentation information, and normal vector of the sample image based on the image features of the sample image. By using a preset loss function, the learned depth information, semantic segmentation information, and normal vector are compared sequentially with the planarized depth image, the semantic segmentation result data, and the normal vector estimation result data. The model parameters of the depth estimation model are adjusted based on the comparison results.

13. The method according to claim 12, characterized in that, The feature learning module includes: a depth estimation module, a semantic segmentation module, and a normal vector estimation module; the feature learning module in the depth estimation model learns the depth information, semantic segmentation information, and normal vectors of the sample image based on the image features of the sample image, including: The depth estimation module in the depth estimation model learns the depth information of the sample image based on its image features. The semantic segmentation module in the depth estimation model learns the semantic segmentation information of the sample image based on its image features. The normal vector of the sample image is learned by the normal vector estimation module in the depth estimation model based on the image features of the sample image.

14. An image depth estimation device, characterized in that, include: The depth estimation unit is used to estimate the depth information of the target image through a depth estimation model to obtain an initial depth image corresponding to the target image. The training data for the depth estimation model includes sample images and planarized depth images corresponding to the sample images; the point cloud data of the first object in the planarized depth image has been pre-planarized; the planarized depth image corresponding to the sample image is obtained through the following steps: acquiring the sample image and the structured light depth image corresponding to the sample image generated by a structured light device; The point cloud data of the first object in the structured light depth image is planarized. The point cloud data of the first object after planarization is mapped back to the structured light depth image, and the mapped structured light depth image is used as the planarized depth image. A pixel localization unit is used to locate problem pixels in the initial depth image; A pixel value adjustment unit is used to adjust the pixel value of the problematic pixel. The result determination unit is used to determine the final depth image corresponding to the target image based on the adjusted initial depth image.

15. An electronic device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions that, when executed by a processor, implement the steps of the method described in any one of claims 1-13.

Images