Image depth estimation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an image depth estimation method and device, electronic equipment and storage medium, wherein the method comprises: obtaining a to-be-processed image and a pre-trained depth estimation model; the depth estimation model is trained according to a sample image, a first depth image corresponding to the sample image and a second depth image corresponding to the sample image; the depth values in the first depth image are collected by a structured light device; in the second depth image, the distances from the collection camera of the sample image to the plane calculated according to the depth values of each pixel point located on the same plane are all equal; the depth estimation model is used to perform depth estimation on the to-be-processed image to obtain a depth image corresponding to the to-be-processed image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image depth estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Depth estimation of images is a crucial component of 3D modeling, and the accuracy of the depth estimation directly impacts the quality of the 3D modeling. Currently, image depth estimation can be performed using deep learning-based neural networks. However, existing depth estimation schemes only learn the mapping relationship between image texture and depth, resulting in poor accuracy of the depth information estimated by neural networks. Summary of the Invention

[0003] This disclosure provides an image depth estimation method, apparatus, electronic device, and storage medium to alleviate the problem of poor accuracy of depth information estimated by neural networks in the prior art, so that the plane can maintain a reasonable structural relationship in the model built using the estimated depth information.

[0004] In a first aspect, embodiments of this disclosure provide an image depth estimation method, including:

[0005] The process involves acquiring an image to be processed and a pre-trained depth estimation model. The depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image. Depth values ​​in the first depth image are acquired using a structured light device. The depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements. These requirements include that the distances from the acquisition camera to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal.

[0006] The depth estimation model is used to estimate the depth of the image to be processed, thereby obtaining the depth image corresponding to the image to be processed.

[0007] In a second aspect, embodiments of this disclosure provide an image depth estimation apparatus, comprising:

[0008] An image acquisition unit is used to acquire an image to be processed and a pre-trained depth estimation model; wherein, the depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image; the depth values ​​in the first depth image are acquired through a structured light device; the depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements; the depth value requirements include: the distances from the acquisition camera of the sample image to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal;

[0009] The depth estimation unit is used to perform depth estimation on the image to be processed using the depth estimation model, so as to obtain a depth image corresponding to the image to be processed.

[0010] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in the first aspect above.

[0011] Fourthly, embodiments of this disclosure provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0012] In one or more embodiments of this disclosure, firstly, an image to be processed and a pre-trained depth estimation model are acquired. Then, the depth of the image to be processed is estimated using the depth estimation model to obtain a depth image corresponding to the image to be processed. The depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image. The depth values ​​in the first depth image are acquired using a structured light device. The depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements. These requirements include that the distances from the acquisition camera to the plane in the sample images, calculated based on the depth values ​​of each pixel located on the same plane, are all equal. Since the distance from the camera to the plane of the sample image obtained from the depth value of each pixel on the same plane in the second depth image is equal, the point cloud data corresponding to each pixel in the model of the plane constructed from the depth value of each pixel on the same plane in the second depth image falls on the plane. This makes the plane relatively flat and able to maintain a reasonable structural relationship. Therefore, when the depth estimation model is trained using the second depth image and the depth estimation model is used to estimate the image depth, the estimated depth information is highly accurate. In the model built using the estimated depth information, the plane can maintain a reasonable structural relationship, thereby alleviating the problems of poor accuracy of depth information estimated by neural networks and the inability of the plane to maintain a reasonable structural relationship in the model built using the estimated depth information in the existing technology. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in one or more embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A schematic flowchart of an image depth estimation method provided in an embodiment of this disclosure;

[0015] Figure 2 This is a schematic diagram of the structure of a depth estimation model provided in an embodiment of the present disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of an image depth estimation device provided in an embodiment of the present disclosure;

[0017] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this disclosure, the technical solutions in one or more embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of the embodiments. Based on one or more embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this disclosure.

[0019] This disclosure provides an image depth estimation method, apparatus, electronic device, and storage medium to alleviate the problems in the prior art where the depth information estimated by neural networks is inaccurate and the planes in the models built using the estimated depth information cannot maintain reasonable structural relationships.

[0020] Figure 1 This is a flowchart illustrating an image depth estimation method provided in an embodiment of the present disclosure, as shown below. Figure 1 As shown, the process includes:

[0021] Step S102: Obtain the image to be processed and a pre-trained depth estimation model; wherein, the depth estimation model is trained based on the sample image, the first depth image corresponding to the sample image, and the second depth image corresponding to the sample image; the depth values ​​in the first depth image are acquired by a structured light device; the depth values ​​of each pixel in the second depth image located on the same plane all meet the preset depth value requirements; the depth value requirements include: the distance from the acquisition camera of the sample image to the plane calculated based on the depth values ​​of each pixel located on the same plane is equal;

[0022] Step S104: The depth of the image to be processed is estimated using a depth estimation model to obtain the depth image corresponding to the image to be processed.

[0023] In one or more embodiments of this disclosure, firstly, an image to be processed and a pre-trained depth estimation model are acquired. Then, the depth of the image to be processed is estimated using the depth estimation model to obtain a depth image corresponding to the image to be processed. The depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image. The depth values ​​in the first depth image are acquired using a structured light device. The depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements. These requirements include that the distances from the acquisition camera to the plane in the sample images, calculated based on the depth values ​​of each pixel located on the same plane, are all equal. Since the distance from the camera to the plane of the sample image obtained from the depth value of each pixel on the same plane in the second depth image is equal, the point cloud data corresponding to each pixel in the model of the plane constructed from the depth value of each pixel on the same plane in the second depth image falls on the plane. This makes the plane relatively flat and able to maintain a reasonable structural relationship. Therefore, when the depth estimation model is trained using the second depth image and the depth estimation model is used to estimate the image depth, the estimated depth information is highly accurate. In the model built using the estimated depth information, the plane can maintain a reasonable structural relationship, thereby alleviating the problems of poor accuracy of depth information estimated by neural networks and the inability of the plane to maintain a reasonable structural relationship in the model built using the estimated depth information in the existing technology.

[0024] Figure 1 The image depth estimation method described below can be applied to a backend server and executed by the backend server. Figure 1 The various steps of the method described.

[0025] In step S102 above, the image to be processed is acquired. This image is the one whose depth is to be estimated, and can be a monocular image captured by a monocular camera, specifically an image of an interior scene of a house. Furthermore, a pre-trained depth estimation model is acquired. The depth estimation model is trained based on sample images, a first depth image corresponding to the sample images, and a second depth image corresponding to the sample images. The sample images can be images collected according to requirements, and can be 2N multiplied by N monocular panoramic images, where N represents the angle. For example, the sample image is a 360-degree multiplied by 180-degree monocular panoramic image. The first depth image is the depth image corresponding to the sample image acquired through a structured light device. The process of acquiring the second depth image and the training process of the depth estimation model will be described in detail later.

[0026] In step S104 above, the image to be processed is input into the depth estimation model. The depth estimation model processes the image and outputs a depth image corresponding to the image to be processed. The image to be processed has the same field of view as the sample image, and can be a monocular panoramic image of 2N multiplied by N, where N represents the angle. For example, the image to be processed is a monocular panoramic image of 360 degrees multiplied by 180 degrees. The depth image corresponding to the image to be processed is an image with the same resolution as the image to be processed. The pixel value of each pixel in the depth image corresponding to the image to be processed represents the depth value of the corresponding pixel in the image to be processed. The depth value of a pixel can be the distance from the camera that captured the image to the pixel (i.e., the camera that acquired the image to be processed).

[0027] In one embodiment, the above depth estimation model is trained through the following steps:

[0028] Acquire a sample image, a first depth map, and a second depth map. Process the sample image using a pre-established depth estimation network to obtain a depth estimation image corresponding to the sample image.

[0029] The parameters of the depth estimation network are adjusted based on the depth estimation image, the first depth map, and the second depth map.

[0030] A depth estimation model is generated based on the parameter-adjusted depth estimation network.

[0031] In this embodiment, a depth estimation network is pre-established, which can be built using a general network structure. A sample image, a first depth map, and a second depth map are acquired. The sample image is then processed through the depth estimation network to obtain a depth estimation image corresponding to the sample image. Then, based on the depth estimation image, the first depth map, and the second depth map, the parameters of the depth estimation network are adjusted. Finally, a depth estimation model is generated based on the parameter-adjusted depth estimation network; for example, the parameter-adjusted depth estimation network can be used as the depth estimation model.

[0032] As can be seen, through this embodiment, the parameters of the depth estimation network can be adjusted based on the depth estimation image, the first depth map, and the second depth map. Then, a depth estimation model is generated based on the depth estimation network with adjusted parameters. This allows the trained depth estimation model to learn the depth information of the first and second depth maps. As a result, when the trained depth estimation model is used to estimate the image depth, the estimated depth information is highly accurate, and the planes in the 3D model built using the estimated depth information can maintain a reasonable structural relationship.

[0033] In one embodiment, the parameters of the depth estimation network are adjusted based on the depth estimation image, a first depth map, and a second depth map, including:

[0034] The difference between the depth estimation image and the first depth map is calculated to obtain the first difference; and the difference between the depth estimation image and the second depth map is calculated to obtain the second difference.

[0035] The parameters of the depth estimation network are adjusted based on the first and second differences.

[0036] Specifically, the depth estimation image is compared with the first depth map using a preset loss function such as the BerHu loss function, and the difference between the depth estimation image and the first depth map is calculated as the first difference. Similarly, the depth estimation image is compared with the second depth map using the same loss function such as the BerHu loss function, and the difference between the depth estimation image and the second depth map is calculated as the second difference.

[0037] Based on the first and second differences, the parameters of the depth estimation network are adjusted in reverse, with the sum between the first and second differences being minimized. After the parameters of the depth estimation network are adjusted, a depth estimation model is generated based on the depth estimation network.

[0038] As can be seen, this embodiment can calculate the difference between the depth estimation image and the first depth map and the difference between the depth estimation image and the second depth map, respectively. Based on these two differences, the parameters of the depth estimation network are adjusted in reverse, so that the trained depth estimation model can better learn the depth information of the first and second depth maps, thereby improving the accuracy of model training.

[0039] As can be seen from the above training process, training the depth estimation model requires sample images, a first depth image, and a second depth image. As mentioned earlier, the sample images can be images acquired according to requirements. The first depth image is the depth image corresponding to the sample image acquired through a structured light device. The second depth image is also the depth image corresponding to the sample image, and the depth value of each pixel in the second depth image located on the same plane meets preset depth value requirements. These requirements include that the distance from the camera that acquired the sample image (i.e., the camera that captured the sample image), calculated based on the depth values ​​of each pixel located on the same plane, to the plane is equal. In one embodiment, the second depth image is obtained through the following steps:

[0040] In the sample image, determine each plane, and, based on the depth value requirement, determine the depth value of each pixel in each plane that meets the depth value requirement;

[0041] A second depth image is generated based on the depth value of each pixel in each plane that meets the depth value requirements.

[0042] First, each plane is identified in the sample image, including ceilings, floors, walls, etc. Then, based on depth requirements, a depth value is determined for each pixel within each plane to meet the requirements. For areas in the sample image other than the planes, the depth value of pixels in those areas can be set to 0. Finally, based on the depth values ​​of each pixel within each plane that meet the depth requirements, a second depth image is generated. In the second depth image, the depth value of each pixel within each plane meets the depth requirements, and the depth value of pixels outside the planes can be set to 0.

[0043] As can be seen, through this embodiment, each plane is first determined in the sample image, and then the depth value of each pixel in each plane that meets the depth value requirement is determined. Then, based on the depth value of each pixel in each plane that meets the depth value requirement, a second depth image is generated. This can ensure that the depth value of each pixel on the same plane in the generated second depth image meets the preset depth value requirement, thereby improving the accuracy of determining the second depth image.

[0044] In one embodiment, determining each plane in a sample image includes:

[0045] Obtain the semantic segmentation image and the floor plan estimation image corresponding to the sample image; the semantic segmentation image is marked with the first planar region of each plane in the sample image; the floor plan estimation image is marked with the second planar region of each plane in the sample image;

[0046] Each plane is determined in the sample image based on its first plane region and its second plane region.

[0047] First, obtain the semantic segmentation image corresponding to the sample image. The semantic segmentation image can determine the first plane region of each plane in the sample image through semantic segmentation. Then, obtain the floor plan estimation image corresponding to the sample image. The floor plan estimation image can determine the second plane region of each plane in the sample image through floor plan estimation. For a plane such as a floor, since its first plane region is determined by semantic segmentation and its second plane region is determined by floor plan estimation, its first plane region and second plane region are not completely the same.

[0048] Then, based on the first plane region and the second plane region of each plane, the region where each plane is located is determined in the sample image, and thus each plane is determined.

[0049] As can be seen, this embodiment can obtain the semantic segmentation image and the floor plan estimation image corresponding to the sample image, and accurately determine the plane by combining the semantic segmentation image and the floor plan estimation image corresponding to the sample image, thereby improving the accuracy of plane determination.

[0050] In one embodiment, determining each plane in the sample image based on a first plane region and a second plane region of each plane includes:

[0051] The intersection of the first planar region and the second planar region of each plane is used to obtain the third planar region of each plane.

[0052] Each plane is determined in the sample image based on the third plane region of each plane.

[0053] Specifically, for each plane, the intersection of its first and second planar regions is taken to obtain its third planar region, thus obtaining the third planar region for each plane. Then, in the sample image, each plane is determined based on its third planar region.

[0054] As can be seen, through this embodiment, for each plane, the intersection of the first plane region and the second plane region of the plane is taken as the third plane region of the plane, thereby ensuring that each plane is accurately determined in the sample image, improving the confidence of the plane and improving the accuracy of the plane.

[0055] As can be seen from the above, determining a plane in a sample image requires the help of the semantic segmentation image and the floor plan estimation image corresponding to the sample image. In one embodiment, the semantic segmentation image corresponding to the sample image is obtained in the following way:

[0056] The sample image is input into the semantic segmentation network, which performs semantic segmentation on the sample image to obtain the semantic segmentation image corresponding to the sample image.

[0057] The semantic segmentation network can be a pre-trained, mature network. It performs semantic segmentation on the sample image to obtain a semantic segmentation image corresponding to the sample image. The semantic segmentation image corresponding to the sample image has the same resolution as the sample image. The pixel value of each pixel in the semantic segmentation image represents the semantic category of the corresponding pixel in the sample image. For example, in the semantic segmentation image, a pixel value of 1 represents the ceiling, a pixel value of 2 represents the floor, and a pixel value of 3 represents the wall. The semantic segmentation image uses different pixel values ​​to represent the first plane region where the ceiling, floor, wall, and other planes in the sample image are located.

[0058] As can be seen, this embodiment enables semantic segmentation of sample images through a semantic segmentation network, resulting in semantically segmented images corresponding to the sample images, thus ensuring the accuracy of obtaining semantically segmented images.

[0059] In one embodiment, the estimated floor plan image corresponding to the sample image is obtained in the following manner:

[0060] The main direction in the sample image is determined using a preset algorithm, and then the main direction is corrected to be parallel to the preset direction.

[0061] The corrected sample image is input into the apartment type estimation network. The apartment type estimation network estimates the apartment type of the corrected sample image to obtain the apartment type estimation image corresponding to the sample image.

[0062] First, a preset algorithm, such as a panoramic vanishing point detection algorithm, is used to determine the main directions in the sample image. The main directions are three mutually perpendicular directions. Then, the detected main directions are corrected to be parallel to the preset directions. When the main directions are three mutually perpendicular directions, each detected main direction is corrected to be parallel to the three preset perpendicular directions.

[0063] Then, the corrected sample image is input into the floor plan estimation network. The floor plan estimation network estimates the floor plan of the corrected sample image to obtain the floor plan estimation image corresponding to the sample image. This floor plan estimation network can be a pre-trained mature network. The floor plan estimation image corresponding to the sample image has the same resolution as the sample image. The floor plan estimation image is marked with the boundary lines between the ceiling and the wall, the boundary lines between the wall and the floor, and the corner points or boundary lines between the walls. The floor plan estimation image represents the second plane region where the ceiling, floor, wall and other planes in the sample image are located by marking this information.

[0064] As can be seen, by first correcting the main direction of the sample image and then using the apartment type estimation network to estimate the apartment type of the corrected sample image, the accuracy of the apartment type estimation can be improved, thereby ensuring the accuracy of obtaining the apartment type estimation image.

[0065] The following example illustrates the process of determining each plane in a sample image.

[0066] First, the vanishing point detection algorithm of the panoramic image is used to obtain the principal direction in the sample image. The principal direction is then corrected to be parallel to the preset principal direction. The principal direction in the sample image refers to the normal direction of the main plane (such as the ceiling, floor, or wall) in the sample image. The main plane can be a plane. Therefore, correcting the principal direction is equivalent to correcting the normal direction of the plane, so that the normal direction of the plane is corrected to be parallel to the preset direction.

[0067] Next, the corrected sample image is input into the trained semantic segmentation network. The semantic segmentation network performs semantic segmentation on the sample image to obtain the semantic segmentation image corresponding to the sample image. In the semantic segmentation image, the different pixel values ​​represent the first plane region where the ceiling, floor, wall and other planes in the sample image are located.

[0068] Furthermore, the corrected sample images are input into the trained floor plan estimation network. The floor plan estimation network performs floor plan estimation on the sample images to obtain the floor plan estimation images corresponding to the sample images. The floor plan estimation images are marked with the boundary lines between the ceiling and the wall, the boundary lines between the wall and the floor, and the corner information or boundary lines between the walls. The floor plan estimation images represent the second plane region where the ceiling, floor, wall and other planes in the sample images are located by marking these information.

[0069] Next, in the semantic segmentation image, the first plane region where the ceiling is located is identified as the first type of ceiling mask, and in the floor plan estimation image, the second plane region where the ceiling is located is identified as the second type of ceiling mask. The intersection of the first type of ceiling mask and the second type of ceiling mask in the semantic segmentation image is used to obtain the third type of ceiling mask. Similarly, in the semantic segmentation image, the first plane region where the floor is located is identified as the first type of floor mask, and in the floor plan estimation image, the second plane region where the floor is located is identified as the second type of floor mask. The intersection of the first type of floor mask and the second type of floor mask in the semantic segmentation image is used to obtain the third type of floor mask.

[0070] Finally, in the semantic segmentation image, the first plane region where the wall is located is determined as the first type of wall mask. The boundary line between the walls is determined based on the corner point information of the boundary between the walls in the floor plan estimation image. Based on the boundary line between the ceiling and the wall, the boundary line between the wall and the floor, and the boundary line between the walls, the second plane region where the wall is located is determined in the floor plan estimation image. The second plane region where the wall is located is determined as the second type of wall mask. The third type of wall mask is obtained by taking the intersection of the first type of wall mask and the second type of wall mask in the semantic segmentation image.

[0071] The third type of ceiling mask, third type of floor mask, and third type of wall mask are the planes in the sample image. It's understandable that the sample image contains one third type of ceiling mask and one third type of floor mask, while there are multiple third type of wall masks. Since the principal directions in the sample image are pre-corrected to be parallel to the preset principal directions, the sample image conforms to the Manhattan hypothesis. Therefore, the third type of ceiling mask, third type of floor mask, and third type of wall mask are respectively regarded as Manhattan planes. In the Manhattan plane, all pixels lie on the same plane. The normal direction of the Manhattan plane is parallel to the three preset principal directions. Specifically, the normal directions of the ceiling and floor are the same principal direction, while the normal direction of the walls is one of the other two principal directions. The Manhattan hypothesis states that in an urban scene, there is a world coordinate system formed by three mutually orthogonal principal directions, and the normal vector of a building plane is parallel to one of the directions in the coordinate system.

[0072] As can be seen, through this embodiment, on the one hand, the main direction in the sample image is pre-corrected to be parallel to the preset main direction, which improves the accuracy of the apartment type estimation; on the other hand, by correcting the main direction of the sample image, each plane determined in the sample image is a Manhattan plane. By training the depth estimation model through the Manhattan plane, the Manhattan plane constraint can be applied to the depth estimation model, so that when the depth estimation model is used to estimate the image depth, the depth information obtained is highly accurate, and the planes in the model established using the estimated depth information can maintain a reasonable structural relationship.

[0073] As described above, after determining each plane in the sample image, it is necessary to determine the depth value of each pixel in each plane that meets the depth value requirement, and then generate a second depth image based on the depth value of each pixel in each plane that meets the depth value requirement. In one embodiment, determining the depth value of each pixel in each plane that meets the depth value requirement includes:

[0074] For each plane, obtain the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction of each pixel;

[0075] For each plane, the depth value that meets the depth requirements for each pixel is determined based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction of each pixel.

[0076] First, for each plane, obtain the pixel coordinates, reference depth value, and normal direction of each pixel in the plane. Then, for each plane, determine the depth value that meets the depth requirements for each pixel in the plane based on the pixel coordinates, reference depth value, and normal direction of each pixel in the plane.

[0077] As can be seen, through this embodiment, for each plane, the depth value of each pixel in the plane that meets the depth value requirement can be determined based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel, thereby accurately determining the depth value of each pixel in the plane that meets the depth value requirement and improving the accuracy of generating the second depth image.

[0078] In one embodiment, determining the depth value of each pixel in the plane that meets the depth requirement based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel includes:

[0079] Based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction of each pixel, determine the distance from the plane corresponding to each pixel to the acquisition camera.

[0080] Based on the depth requirements and the distance from the plane to the acquisition camera corresponding to each pixel in the plane, determine the depth value of each pixel in the plane that meets the depth requirements.

[0081] For each plane, based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction of each pixel, the distance from the plane corresponding to each pixel in the plane to the camera capturing the sample image is determined. Each pixel in the plane corresponds to a distance from one plane to the camera. Next, for each plane, based on the depth requirement and the distance from the plane to the camera corresponding to each pixel in the plane, the depth value that meets the depth requirement is determined.

[0082] As can be seen, this embodiment can improve the accuracy of determining the depth value of a pixel by first determining the distance from the plane corresponding to each pixel to the camera that acquires the sample image, and then determining the depth value of each pixel.

[0083] In one embodiment, determining the distance from the plane corresponding to each pixel to the acquisition camera based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel includes:

[0084] The three-dimensional coordinates of each pixel in the plane are determined based on the pixel coordinates and the reference depth value of each pixel.

[0085] Based on the 3D coordinates of each pixel in the plane and the normal direction corresponding to each pixel, the distance from the plane corresponding to each pixel to the acquisition camera is determined. This acquisition camera is the camera used to acquire the sample images.

[0086] Specifically, for any plane, the three-dimensional coordinates of each pixel in the plane are determined based on the pixel coordinates and the reference depth value of each pixel. Let p be the pixel coordinates of any pixel in any plane. i (u,v), the reference depth value of this pixel is d(p i Then, based on the panoramic camera model, the three-dimensional coordinates of the pixel in space can be calculated as P. i (x, y, z). Where i is a positive integer, and the calculation formula for the panoramic camera model is:

[0087] P i (x,y,z)=d(p i )*T(p i (u,v)); where T represents the fixed transformation relationship from the two-dimensional coordinate system to the three-dimensional coordinate system under the panoramic camera model.

[0088] Then, for any plane, based on the 3D coordinates of each pixel in the plane and the normal direction corresponding to each pixel, determine the distance from the plane corresponding to each pixel to the camera capturing the sample image. Here, one pixel corresponds to one distance from the plane to the camera capturing the sample image. The distance from the plane corresponding to any pixel to the camera capturing the sample image can be calculated using the following formula:

[0089] Wherein, D(p) i ) represents pixel p i The distance P from the corresponding plane to the camera that acquired the sample image i For this pixel p i Corresponding to three-dimensional coordinates in space, For pixel p i The corresponding normal direction, this formula is the plane equation.

[0090] It is understood that for any pixel on any plane, the normal direction corresponding to that pixel can be the normal direction of the plane containing that pixel. In the previous embodiment, if the principal direction of the sample image is first corrected to be parallel to a preset direction, then the normal direction of the plane containing the pixel is the preset direction (the normal direction of the plane, as the principal direction, has been corrected to be parallel to the preset direction), that is, the normal direction corresponding to the pixel is the aforementioned preset direction. Of course, for any pixel on any plane, the normal direction corresponding to that pixel can also be determined by other methods.

[0091] As can be seen, through this embodiment, for any pixel on any plane, the three-dimensional coordinates of the pixel can be calculated first, and then the distance from the plane corresponding to the pixel to the acquisition camera can be calculated based on the three-dimensional coordinates. The calculation efficiency is high and the calculation results are accurate.

[0092] In one embodiment, determining the depth value of each pixel in the plane that meets the depth requirement, based on the depth value requirement and the distance from the plane to the acquisition camera corresponding to each pixel in the plane, includes:

[0093] Based on the depth requirement, the average distance is obtained by taking the average distance from the plane corresponding to each pixel in the plane to the acquisition camera;

[0094] For each pixel in the plane, the depth value that meets the depth requirements is determined based on the average distance, the normal direction corresponding to the pixel, and the pixel coordinates.

[0095] Specifically, for any plane, the average distance from the plane to the acquisition camera corresponding to each pixel in the plane is taken to obtain the average distance. Then, based on the average distance, the normal direction corresponding to the pixel and the pixel coordinates, the depth value of the pixel in the plane that meets the depth value requirement is determined.

[0096] For any plane, the average distance is obtained by taking the average distance from the plane corresponding to each pixel in the plane to the acquisition camera using the following formula;

[0097]

[0098] Among them, D j This is the average distance, where N is the number of pixels in the plane, and i is a positive integer starting from 1.

[0099] Then, based on the average distance, the normal direction corresponding to the pixel, and the pixel coordinates, the depth value of the pixel that meets the depth value requirements is obtained:

[0100]

[0101] in, For pixel p i The depth value that meets the depth requirements. For pixel p i The corresponding normal direction, T represents the fixed transformation relationship from the 2D coordinate system to the 3D coordinate system in the panoramic camera model, p i (u,v) are pixel coordinates.

[0102] Since for any given plane, the average distance from the plane to the camera corresponding to each pixel in that plane is taken, and the depth value of the pixel that meets the depth requirement is deduced from the average value, it can be guaranteed that the depth value of each pixel in that plane meets the preset depth requirement. That is, the distance from the camera to the plane of the sample image calculated based on the depth value of each pixel on that plane is equal, and is always the average distance D. j .

[0103] As can be seen, through this embodiment, for any plane, the average distance from the plane to the acquisition camera corresponding to each pixel in the plane is first taken according to the depth value requirement, and then for each pixel in the plane, the depth value that meets the depth value requirement is deduced based on the average distance, thereby ensuring that the depth value of each pixel in the plane meets the preset depth value requirement. That is, the distance from the acquisition camera to the plane of the sample image calculated based on the depth value of each pixel in the plane is equal, thereby accurately generating the second depth image.

[0104] As described above, the planes in the sample image are determined based on the corresponding semantic segmentation image and the corresponding floor plan estimation image. The semantic segmentation image is obtained through a semantic segmentation network. The floor plan estimation image is obtained through a floor plan estimation network. The depth value of each pixel in each plane of the sample image that meets the depth requirements is determined based on the reference depth value of each pixel.

[0105] In one embodiment, the semantic segmentation network and the floor plan estimation network are located in the depth estimation model; the reference depth value comes from the depth estimation image processed by the depth estimation network, or from the first depth image.

[0106] Specifically, the semantic segmentation network and the apartment layout estimation network can be pre-trained, high-accuracy models. These two networks can be set in the deep estimation model. During the training phase of the deep estimation model, while training the deep estimation network, the semantic segmentation network and the apartment layout estimation network can be further trained to further improve the accuracy of the network.

[0107] In addition, in the aforementioned process of calculating the depth value that meets the depth value requirements, the reference depth value of the pixel used can be the depth value in the depth estimation image obtained by the depth estimation network processing the sample image during the training process of the depth estimation network, or it can be the depth value in the first depth image.

[0108] As can be seen, in this embodiment, by incorporating the semantic segmentation network and the floor plan estimation network into the depth estimation model, the semantic segmentation network and the floor plan estimation network can be trained simultaneously during the training phase of the depth estimation model, thereby further improving the network's accuracy. By setting the reference depth value to the depth estimation image obtained by the depth estimation network processing the sample image, the effect of generating a second depth image can be achieved while training the depth estimation network.

[0109] In one scenario, if the semantic segmentation network and the floor plan estimation network are independent networks not located in the depth estimation model, and the reference depth value comes from the first depth image, then the second depth image can be generated before training the depth estimation network.

[0110] In another scenario, if the semantic segmentation network and the floor plan estimation network are located in the depth estimation model, and / or the reference depth value comes from the depth estimation image obtained by the depth estimation network processing the sample image, then it is possible to generate a second depth image while training the depth estimation network.

[0111] In one embodiment, a semantic segmentation network and a floor plan estimation network are incorporated into the depth estimation model. Accordingly, during the training of the depth estimation model, after acquiring the sample images, the method further includes:

[0112] The sample image is processed by a semantic segmentation network to obtain the semantic segmentation image corresponding to the sample image, and the sample image is processed by a floor plan estimation network to obtain the floor plan estimation image corresponding to the sample image.

[0113] The parameters of the semantic segmentation network and the floor plan estimation network are adjusted based on the semantic segmentation image, the floor plan estimation image, the ground truth image of the semantic segmentation corresponding to the sample image, and the ground truth image of the floor plan estimation corresponding to the sample image.

[0114] Specifically, the semantic segmentation network and the apartment type estimation network are set in the depth estimation model. While training the depth estimation network in the depth estimation model, a second depth image is generated. Furthermore, the semantic segmentation network and the apartment type estimation network are further optimized and trained.

[0115] The semantic segmentation network and the floor plan estimation network are pre-trained networks with a certain level of accuracy. The further optimization and training process for these networks involves: processing the sample image using the semantic segmentation network to obtain the corresponding semantic segmentation image, which is the semantic segmentation image used to generate the second depth image; and processing the sample image using the floor plan estimation network to obtain the corresponding floor plan estimation image, which is the floor plan estimation image used to generate the second depth image. The parameters of the semantic segmentation network and the floor plan estimation network are adjusted based on the semantic segmentation image, the floor plan estimation image, the ground truth semantic segmentation image corresponding to the sample image, and the ground truth floor plan estimation image corresponding to the sample image. Specifically, the ground truth semantic segmentation image corresponding to the sample image is the accurate semantic segmentation result obtained beforehand, and the ground truth floor plan estimation image corresponding to the sample image is the accurate floor plan estimation result obtained beforehand.

[0116] As can be seen, through this embodiment, by setting the semantic segmentation network and the floor plan estimation network in the depth estimation model, a second depth image can be generated simultaneously during the training phase of the depth estimation model, and the depth estimation network can be trained simultaneously. Furthermore, by simultaneously combining the ground truth image of semantic segmentation corresponding to the sample image and the ground truth image of floor plan estimation corresponding to the sample image, the semantic segmentation network and the floor plan estimation network can be trained further, thereby improving the accuracy of the semantic segmentation network and the floor plan estimation network, and enriching the functionality of the trained depth estimation model, so that the trained depth estimation model can also be used for semantic segmentation and floor plan estimation.

[0117] In one embodiment, the parameters of the semantic segmentation network and the floor plan estimation network are adjusted based on the semantic segmentation image, the floor plan estimation image, the ground truth semantic segmentation image corresponding to the sample image, and the ground truth floor plan estimation image corresponding to the sample image, including:

[0118] The difference between the semantic segmentation image and the ground truth semantic segmentation image corresponding to the sample image is calculated to obtain the third difference. The difference between the floor plan estimation image and the ground truth floor plan estimation image corresponding to the sample image is calculated to obtain the fourth difference.

[0119] The parameters of the semantic segmentation network and the apartment type estimation network are adjusted based on the third and fourth differences.

[0120] Specifically, the semantic segmentation image is compared with the ground truth semantic segmentation image corresponding to the sample image using a preset loss function such as the BerHu loss function. The difference between the semantic segmentation image and the ground truth semantic segmentation image corresponding to the sample image is calculated as the third difference. Similarly, the floor plan estimation image is compared with the ground truth floor plan estimation image corresponding to the sample image using the same loss function such as the BerHu loss function. The difference between the floor plan estimation image and the ground truth floor plan estimation image corresponding to the sample image is calculated as the fourth difference. Based on the third difference, the parameters of the semantic segmentation network are adjusted in reverse when the third difference is minimized. Based on the fourth difference, the parameters of the floor plan estimation network are adjusted in reverse when the fourth difference is minimized.

[0121] As can be seen, through this embodiment, the third difference between the semantic segmentation image and the ground truth image of the semantic segmentation corresponding to the sample image, and the fourth difference between the floor plan estimation image and the ground truth image of the floor plan estimation corresponding to the sample image can be calculated respectively. Based on the third difference and the fourth difference, the parameters of the semantic segmentation network and the floor plan estimation network are adjusted respectively, thereby further training the semantic segmentation network and the floor plan estimation network and enriching the function of the deep estimation model.

[0122] In one embodiment, the depth estimation model includes a depth estimation network, a semantic segmentation network, a floor plan estimation network, and a feature extraction network. The depth estimation network, semantic segmentation network, and floor plan estimation network share the feature extraction network. The feature extraction network is used to extract image features from the sample images. Based on this, the depth estimation network obtains a depth-estimated image based on the image features, the semantic segmentation network obtains a semantic segmentation image based on the image features, and the floor plan estimation network obtains a floor plan estimation image based on the image features.

[0123] Specifically, a feature extraction network shared by the depth estimation network, semantic segmentation network, and floor plan estimation network is incorporated into the depth estimation model. This feature extraction network can be a pre-trained network with a certain level of accuracy. The feature extraction network is used to extract image features from the sample images. The depth estimation network obtains the depth estimation image based on the image features, the semantic segmentation network obtains the semantic segmentation image based on the image features, and the floor plan estimation network obtains the floor plan estimation image based on the image features.

[0124] As can be seen, in this embodiment, since the depth estimation network, semantic segmentation network, and apartment type estimation network share the feature extraction network, the feature extraction network can also be trained when training the depth estimation model. This allows the feature extraction performance of the feature extraction network to meet the requirements of multiple subsequent processing steps such as depth estimation, semantic segmentation, and apartment type estimation, thereby improving the feature extraction capability of the feature extraction network and enhancing the computational capability of the trained depth estimation model.

[0125] Figure 2This is a schematic diagram of the structure of a depth estimation model provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, the depth estimation model includes: a feature extraction network, a depth estimation network, a semantic segmentation network, and a floor plan estimation network. During the training of the depth estimation model, each network within the model is trained simultaneously, and a second depth image is generated. Figure 2 The training process for the depth estimation model in the text is as follows:

[0126] 1. Obtain the sample image, the first depth image, the semantic segmentation ground image corresponding to the sample image, and the floor plan estimation ground image corresponding to the sample image. Input the sample image, the first depth image, the semantic segmentation ground image, and the floor plan estimation ground image into the depth estimation model.

[0127] 2. Extract image features from sample images using a feature extraction network;

[0128] 3. Generate depth-estimated images based on image features using a depth estimation network;

[0129] 4. Generate semantically segmented images based on image features using a semantic segmentation network;

[0130] 5. Generate estimated floor plan images based on image features using a floor plan estimation network;

[0131] 6. Using semantic segmentation images and floor plan estimation images, determine each plane in the sample images;

[0132] 7. Using the depth values ​​in the depth estimation image as reference depth values, determine the depth value that meets the depth value requirements for each pixel in each plane;

[0133] 8. Generate a second depth image based on the depth value of each pixel in each plane that meets the depth value requirements;

[0134] 9. Calculate the difference between the depth estimation image and the first depth map to obtain the first difference; calculate the difference between the depth estimation image and the second depth map to obtain the second difference; adjust the parameters of the depth estimation network based on the first and second differences.

[0135] 10. Calculate the difference between the semantic segmentation image and the semantic segmentation ground truth image to obtain the third difference. Based on the third difference, adjust the parameters of the semantic segmentation network accordingly.

[0136] 11. Calculate the difference between the estimated floor plan image and the true floor plan image to obtain the fourth difference value. Based on the fourth difference value, adjust the parameters of the floor plan estimation network.

[0137] Figure 2 For the sake of simplicity, only the necessary network is shown; details such as the loss function are not illustrated. Figure 2 When deploying the model online, the semantic segmentation network and the apartment type estimation network can be removed to ensure the speed of online depth estimation.

[0138] As can be seen, through the embodiments of this disclosure, when training the depth estimation model, the semantic segmentation network and the floor plan estimation network can be used to obtain Manhattan plane semantics, and more structured semantic information can be mined from a single image. The Manhattan plane constraint is applied to the network to estimate the depth, which helps the network to learn more fully the geometric structure information contained in the indoor depth from the limited training data, enhances the monocular panoramic depth estimation model capability, and reduces the amount of training data required to train the depth estimation model.

[0139] Figure 3 This is a schematic diagram of the structure of an image depth estimation device provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, the device includes:

[0140] Image acquisition unit 31 is used to acquire the image to be processed and a pre-trained depth estimation model; wherein, the depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image; the depth values ​​in the first depth image are acquired by a structured light device; the depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements; the depth value requirements include: the distances from the acquisition camera of the sample image to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal;

[0141] The depth estimation unit 32 is used to perform depth estimation on the image to be processed using the depth estimation model to obtain a depth image corresponding to the image to be processed.

[0142] Optionally, the device further includes a model training unit for:

[0143] The sample image is acquired, and the sample image is processed through a pre-established depth estimation network to obtain the depth estimation image corresponding to the sample image;

[0144] The parameters of the depth estimation network are adjusted based on the depth estimation image, the first depth map, and the second depth map.

[0145] The depth estimation model is generated based on the depth estimation network after parameter adjustment.

[0146] Optionally, the model training unit is specifically used for:

[0147] Calculate the difference between the depth estimation image and the first depth map to obtain a first difference, and calculate the difference between the depth estimation image and the second depth map to obtain a second difference;

[0148] The parameters of the depth estimation network are adjusted based on the first difference and the second difference.

[0149] Optionally, the model training unit is further specifically used for:

[0150] Each plane is determined in the sample image, and a depth value that meets the depth value requirement is determined for each pixel in each plane according to the depth value requirement;

[0151] The second depth image is generated based on the depth value of each pixel in each plane that meets the depth value requirement.

[0152] Optionally, the model training unit is further specifically used for:

[0153] Obtain the semantic segmentation image corresponding to the sample image and the floor plan estimation image corresponding to the sample image; the semantic segmentation image is marked with a first planar region for each plane in the sample image; the floor plan estimation image is marked with a second planar region for each plane in the sample image;

[0154] Each plane is determined in the sample image based on the first plane region and the second plane region of each plane.

[0155] Optionally, the model training unit is further specifically used for:

[0156] The intersection of the first planar region and the second planar region of each plane is taken to obtain the third planar region of each plane;

[0157] Each plane is determined in the sample image based on the third plane region of each plane.

[0158] Optionally, the model training unit is further specifically used for:

[0159] The sample image is input into a semantic segmentation network, and the semantic segmentation network performs semantic segmentation on the sample image to obtain a semantic segmentation image corresponding to the sample image.

[0160] Optionally, the model training unit is further specifically used for:

[0161] The main direction in the sample image is determined using a preset algorithm, and the main direction is corrected to be parallel to the preset direction;

[0162] The corrected sample image is input into the apartment type estimation network, and the apartment type estimation network is used to estimate the apartment type of the corrected sample image to obtain the apartment type estimation image corresponding to the sample image.

[0163] Optionally, the model training unit is further specifically used for:

[0164] For each plane, obtain the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel;

[0165] For each plane, the depth value of each pixel in the plane that meets the depth value requirement is determined based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel.

[0166] Optionally, the model training unit is further specifically used for:

[0167] Based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel, the distance from the plane to the acquisition camera corresponding to each pixel in the plane is determined.

[0168] Based on the depth requirement and the distance from the plane to the acquisition camera corresponding to each pixel in the plane, the depth value of each pixel in the plane that meets the depth requirement is determined.

[0169] Optionally, the model training unit is further specifically used for:

[0170] The three-dimensional coordinates of each pixel in the plane are determined based on the pixel coordinates and the reference depth value of each pixel.

[0171] Based on the three-dimensional coordinates of each pixel in the plane and the normal direction corresponding to each pixel in the plane, the distance from the plane to the acquisition camera corresponding to each pixel in the plane is determined.

[0172] Optionally, the model training unit is further specifically used for:

[0173] Based on the depth requirement, the average distance is obtained by taking the average distance from the plane to the acquisition camera corresponding to each pixel in the plane;

[0174] For each pixel in the plane, the depth value that meets the depth requirement is determined based on the average distance, the normal direction corresponding to the pixel, and the pixel coordinates of the pixel.

[0175] Optionally, the plane is determined based on the semantic segmentation image corresponding to the sample image and the floor plan estimation image corresponding to the sample image; the semantic segmentation image is obtained through a semantic segmentation network; the floor plan estimation image is obtained through a floor plan estimation network; the depth value of each pixel in each plane that meets the depth value requirement is determined based on the reference depth value of each pixel.

[0176] The semantic segmentation network and the apartment type estimation network are located in the depth estimation model; the reference depth value comes from the depth estimation image processed by the depth estimation network, or from the first depth image.

[0177] Optionally, the model training unit is further specifically used for:

[0178] The sample image is processed by the semantic segmentation network to obtain the semantic segmentation image corresponding to the sample image; the sample image is processed by the floor plan estimation network to obtain the floor plan estimation image corresponding to the sample image.

[0179] The parameters of the semantic segmentation network and the floor plan estimation network are adjusted based on the semantic segmentation image, the floor plan estimation image, the ground truth image of the semantic segmentation corresponding to the sample image, and the ground truth image of the floor plan estimation corresponding to the sample image.

[0180] Optionally, the model training unit is further specifically used for:

[0181] The difference between the semantic segmentation image and the ground truth image corresponding to the sample image is calculated to obtain the third difference; the difference between the floor plan estimation image and the ground truth image corresponding to the sample image is calculated to obtain the fourth difference.

[0182] Based on the third and fourth differences, the parameters of the semantic segmentation network and the apartment type estimation network are adjusted accordingly.

[0183] Optionally, in the depth estimation model, the depth estimation network, the semantic segmentation network, and the apartment layout estimation network share a feature extraction network; the feature extraction network is used to extract image features from the sample images.

[0184] The depth estimation network obtains the depth estimation image based on the image features; the semantic segmentation network obtains the semantic segmentation image based on the image features; and the floor plan estimation network obtains the floor plan estimation image based on the image features.

[0185] The image depth estimation device in this embodiment can implement the various processes of the above-described image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0186] One embodiment of this disclosure also provides an electronic device. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 401 and memories 402, with the memory 402 storing one or more application programs or data. The memory 402 can be temporary or persistent storage. The application programs stored in the memory 402 may include one or more modules (not shown), each module including a series of computer-executable instructions within the electronic device. Furthermore, the processor 401 may be configured to communicate with the memory 402, executing the series of computer-executable instructions stored in the memory 402 on the electronic device. The electronic device may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input or output interfaces 405, one or more keyboards 406, etc.

[0187] In one specific embodiment, the electronic device is an image depth estimation device, specifically a backend server, including a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following process:

[0188] The process involves acquiring an image to be processed and a pre-trained depth estimation model. The depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image. Depth values ​​in the first depth image are acquired using a structured light device. The depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements. These requirements include that the distances from the acquisition camera to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal.

[0189] The depth estimation model is used to estimate the depth of the image to be processed, thereby obtaining the depth image corresponding to the image to be processed.

[0190] The image depth estimation device in this embodiment can implement the various processes of the above-described image depth estimation method embodiments and achieve the same effect and function, which will not be repeated here.

[0191] Another embodiment of this disclosure also provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process:

[0192] The process involves acquiring an image to be processed and a pre-trained depth estimation model. The depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image. Depth values ​​in the first depth image are acquired using a structured light device. The depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements. These requirements include that the distances from the acquisition camera to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal.

[0193] The depth estimation model is used to estimate the depth of the image to be processed, thereby obtaining the depth image corresponding to the image to be processed.

[0194] The storage medium in this embodiment can implement the various processes of the above-described image depth estimation method embodiments and achieve the same effects and functions, which will not be repeated here.

[0195] In various embodiments of this disclosure, the computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0196] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0197] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0198] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0199] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing the embodiments of this disclosure, the functions of each unit can be implemented in one or more software and / or hardware.

[0200] Those skilled in the art will understand that one or more embodiments of this disclosure can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0205] One or more embodiments of this disclosure can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0206] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0207] The above description is merely an embodiment of this disclosure and is not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. An image depth estimation method, characterized in that, include: The process involves acquiring an image to be processed and a pre-trained depth estimation model. The depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image. Depth values ​​in the first depth image are acquired using a structured light device. The depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements. These requirements include that the distances from the acquisition camera to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal. The depth estimation model is used to estimate the depth of the image to be processed, thereby obtaining the depth image corresponding to the image to be processed.

2. The method according to claim 1, characterized in that, The depth estimation model is obtained through the following steps; The sample image is acquired, and the sample image is processed through a pre-established depth estimation network to obtain the depth estimation image corresponding to the sample image; The parameters of the depth estimation network are adjusted based on the depth estimation image, the first depth map, and the second depth map. The depth estimation model is generated based on the depth estimation network after parameter adjustment.

3. The method according to claim 2, characterized in that, The step of adjusting the parameters of the depth estimation network based on the depth estimation image, the first depth map, and the second depth map includes: Calculate the difference between the depth estimation image and the first depth map to obtain a first difference, and calculate the difference between the depth estimation image and the second depth map to obtain a second difference; The parameters of the depth estimation network are adjusted based on the first difference and the second difference.

4. The method according to claim 2, characterized in that, The second depth image is obtained through the following steps: Each plane is determined in the sample image, and a depth value that meets the depth value requirement is determined for each pixel in each plane according to the depth value requirement; The second depth image is generated based on the depth value of each pixel in each plane that meets the depth value requirement.

5. The method according to claim 4, characterized in that, Determining each plane in the sample image includes: Obtain the semantic segmentation image corresponding to the sample image and the floor plan estimation image corresponding to the sample image; the semantic segmentation image is marked with a first planar region for each plane in the sample image; the floor plan estimation image is marked with a second planar region for each plane in the sample image; Each plane is determined in the sample image based on the first plane region and the second plane region of each plane.

6. The method according to claim 5, characterized in that, Determining each plane in the sample image based on the first plane region and the second plane region of each plane includes: The intersection of the first planar region and the second planar region of each plane is taken to obtain the third planar region of each plane; Each plane is determined in the sample image based on the third plane region of each plane.

7. The method according to claim 5, characterized in that, The semantic segmentation image corresponding to the sample image is obtained through the following method: The sample image is input into a semantic segmentation network, and the semantic segmentation network performs semantic segmentation on the sample image to obtain a semantic segmentation image corresponding to the sample image.

8. The method according to claim 5, characterized in that, The estimated floor plan image corresponding to the sample image is obtained using the following method: The main direction in the sample image is determined using a preset algorithm, and the main direction is corrected to be parallel to the preset direction; The corrected sample image is input into the apartment type estimation network, and the apartment type estimation network is used to estimate the apartment type of the corrected sample image to obtain the apartment type estimation image corresponding to the sample image.

9. The method according to claim 4, characterized in that, The step of determining the depth value of each pixel in each plane that meets the depth value requirement according to the depth value requirement includes: For each plane, obtain the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel; For each plane, the depth value of each pixel in the plane that meets the depth value requirement is determined based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel.

10. The method according to claim 9, characterized in that, The step of determining the depth value of each pixel in the plane that meets the depth value requirement based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel includes: Based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel, the distance from the plane to the acquisition camera corresponding to each pixel in the plane is determined. Based on the depth requirement and the distance from the plane to the acquisition camera corresponding to each pixel in the plane, the depth value of each pixel in the plane that meets the depth requirement is determined.

11. The method according to claim 10, characterized in that, The step of determining the distance from the plane to the acquisition camera corresponding to each pixel in the plane based on the pixel coordinates of each pixel in the plane, the reference depth value of each pixel, and the normal direction corresponding to each pixel includes: The three-dimensional coordinates of each pixel in the plane are determined based on the pixel coordinates and the reference depth value of each pixel. Based on the three-dimensional coordinates of each pixel in the plane and the normal direction corresponding to each pixel in the plane, the distance from the plane to the acquisition camera corresponding to each pixel in the plane is determined.

12. The method according to claim 10, characterized in that, The step of determining the depth value of each pixel in the plane that meets the depth requirement based on the depth value requirement and the distance from the plane to the acquisition camera corresponding to each pixel in the plane includes: Based on the depth requirement, the average distance is obtained by taking the average distance from the plane to the acquisition camera corresponding to each pixel in the plane; For each pixel in the plane, the depth value that meets the depth requirement is determined based on the average distance, the normal direction corresponding to the pixel, and the pixel coordinates of the pixel.

13. The method according to claim 4, characterized in that, The plane is determined based on the semantic segmentation image corresponding to the sample image and the floor plan estimation image corresponding to the sample image; the semantic segmentation image is obtained through a semantic segmentation network. The floor plan estimation image is obtained through a floor plan estimation network; the depth value of each pixel in each plane that meets the depth value requirement is determined based on the reference depth value of each pixel. The semantic segmentation network and the apartment layout estimation network are located in the depth estimation model; The reference depth value comes from the depth estimation image obtained by the depth estimation network, or from the first depth image.

14. The method according to claim 13, characterized in that, After acquiring the sample image, the method further includes: The sample image is processed by the semantic segmentation network to obtain the semantic segmentation image corresponding to the sample image; the sample image is processed by the floor plan estimation network to obtain the floor plan estimation image corresponding to the sample image. The parameters of the semantic segmentation network and the floor plan estimation network are adjusted based on the semantic segmentation image, the floor plan estimation image, the ground truth image of the semantic segmentation corresponding to the sample image, and the ground truth image of the floor plan estimation corresponding to the sample image.

15. The method according to claim 14, characterized in that, The step of adjusting the parameters of the semantic segmentation network and the floor plan estimation network based on the semantic segmentation image, the floor plan estimation image, the ground truth image of semantic segmentation corresponding to the sample image, and the ground truth image of floor plan estimation corresponding to the sample image includes: The difference between the semantic segmentation image and the semantic segmentation ground truth image corresponding to the sample image is calculated to obtain the third difference; the difference between the floor plan estimation image and the floor plan estimation ground truth image corresponding to the sample image is calculated to obtain the fourth difference. Based on the third and fourth differences, the parameters of the semantic segmentation network and the apartment type estimation network are adjusted accordingly.

16. The method according to claim 15, characterized in that, In the depth estimation model, the depth estimation network, the semantic segmentation network, and the apartment layout estimation network share a common feature extraction network; The feature extraction network is used to extract image features from the sample images; The depth estimation network obtains the depth-estimated image based on the image features; the semantic segmentation network obtains the semantic segmentation image based on the image features; and the floor plan estimation network obtains the floor plan estimation image based on the image features.

17. An image depth estimation device, characterized in that, include: An image acquisition unit is used to acquire an image to be processed and a pre-trained depth estimation model; wherein, the depth estimation model is trained based on a sample image, a first depth image corresponding to the sample image, and a second depth image corresponding to the sample image; the depth values ​​in the first depth image are acquired through a structured light device; the depth values ​​of each pixel in the second depth image located on the same plane all meet preset depth value requirements; the depth value requirements include: the distances from the acquisition camera of the sample image to the plane, calculated based on the depth values ​​of each pixel located on the same plane, are all equal; The depth estimation unit is used to perform depth estimation on the image to be processed using the depth estimation model, so as to obtain a depth image corresponding to the image to be processed.

18. An electronic device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions that, when executed by a processor, implement the steps of the method described in any one of claims 1-16.

Images