Indoor scene 3D layout estimation method and device based on corner point depth prediction

Abstract

The embodiments of the present disclosure provide a method and device for estimating the 3D layout of indoor scenes based on corner point depth prediction, which is applied to the field of three-dimensional layout technology of indoor scenes. The method comprises acquiring a target indoor scene image; inputting the target indoor scene image into a deep learning model, outputting a predicted corner point heat map, corner point depth value and embedding vector; generating corner point coordinates according to the corner point heat map; clustering the embedding vectors to obtain a rough segmentation map, and using each plane area of ​​the rough segmentation map to perform dilation and corrosion operations to obtain a target area; determining the target corner point according to the corner point coordinates and the target area; performing plane fitting according to the corner point depth value of the target corner point to obtain a target depth map corresponding to each plane; performing depth map intersection calculation on the target depth map to obtain a layout depth map. A more accurate indoor layout depth map is obtained without being affected by position changes caused by cropping.

Description

Technical Field

[0001] The present disclosure relates to the technical field of three-dimensional layout of indoor scenes, and in particular to a method and device for estimating a 3D layout of an indoor scene based on corner point depth prediction. Background Art

[0002] 3D layout estimation of indoor scenes has always been an important research topic in the field of computer vision, and is of great significance for applications such as robot navigation, scene reconstruction, and virtual reality. The task of indoor layout estimation is to locate and identify layout components (floor, ceiling, and walls) from one or more views and infer their 3D geometry.

[0003] In the task of indoor layout estimation, traditional layout methods mainly rely on low-level features, such as local color, image edge information, etc., combined with vanishing point detection, and post-processing to assist in processing abnormal features, and finally generate candidate layouts. For example, through vanishing point detection and ray sampling from the vanishing point to the edge of the image, a candidate layout is generated, and then an evaluation function is trained using low-level image features to evaluate the layout, and finally the layout with the highest score is selected as the best layout. Many subsequent researchers followed this entire process to improve the quality of candidate layouts, optimize the efficiency of ray sampling, and ensure the rationality of the selected evaluation function. However, since the image features used by such methods are often low-level local features, lacking semantic information and global expression capabilities, the accuracy of such methods is not ideal, and the computational complexity is also high.

[0004] Recent deep learning-based indoor layout estimation methods can be divided into two categories: semantic segmentation methods for ceilings, floors, and walls, and key point detection methods. For example, a fully convolutional network (FCN) is used to predict edge information of indoor images, and a new vanishing line sampling method is proposed. Specifically, the segmentation masks of the ceiling, floor, and wall are predicted using FCN, and then the initial vanishing line positions are obtained using logistic regression, followed by greedy iterations to continuously update the positions of the vanishing points and the angles of the vanishing lines until the layout that best matches the predicted edge map is obtained. Another example is a coarse-to-fine indoor layout estimation method. The method is divided into two stages. The first stage uses FCN to predict the edge map, and the second stage obtains a finer layout through geometric constraints including layout contour straightness, surface smoothness, and layout detail refinement. For another example, the indoor layout estimation task is regarded as a key point positioning problem, and an encoder-decoder structure is proposed. The network predicts the room type and the position of the key points respectively, and then uses the results of the predicted room type to filter the key corner point heat map to generate the final indoor layout. For example, when using three segmentation hypotheses to estimate the room layout, first calculate the segmentation based on each hypothesis, then predict the layout defined by 2D key points, and finally compare the layout generated from the 2D key points with the corresponding image segmentation to select the best matching layout. The above method can only estimate the layout of box-type houses. Recently, the use of deep neural networks for 3D layout estimation of indoor scenes has become increasingly active. For example, by predicting pixel-level surface parameters, learning to infer the depth map of the dominant plane in the scene, and generating the layout through the intersection of the depth map. A multi-branch network is proposed to simultaneously detect planes and vertical lines between adjacent walls, as well as the 3D parameters of each plane, and then use geometric reasoning methods to achieve room layout.

[0005] In summary, thanks to deep learning, indoor layout estimation methods have made great progress. Most methods mainly restore the layout by learning the depth parameters or plane equations of the main planes through the network. Although the performance has been significantly improved using existing methods, the depth parameters or plane equations of a plane in an image are actually determined by its appearance and position in the image, rather than translation invariant. It is difficult for general convolutional networks to fully understand the specific position information of each element in the image, which brings difficulties to the task of indoor layout estimation. For example, the original image is cropped twice differently. For point P in the original image, its position is obviously different in the two images after different cropping, but their depth values ​​are the same, which leads to completely different output parameters. However, translation invariance is an important property in image processing tasks because the position of the object in the image may change.

[0006] Therefore, it is necessary to design a neural network that can recognize the content of the object without being affected by the location, so as to obtain a more accurate depth map of the indoor layout. Summary of the invention

[0007] The present disclosure provides a method and device for estimating 3D layout of an indoor scene based on corner point depth prediction.

[0008] According to a first aspect of the present disclosure, a method for estimating 3D layout of an indoor scene based on corner depth prediction is provided. The method comprises:

[0009] Acquire the target indoor scene image;

[0010] Input the target indoor scene image into a deep learning model, and output a predicted corner point heat map, corner point depth value, and embedding vector;

[0011] Generate corner point coordinates according to the corner point heat map;

[0012] Clustering the embedding vectors to obtain a rough segmentation map, and performing dilation and erosion operations on each plane region of the rough segmentation map to obtain a target region;

[0013] Determine a target corner point according to the corner point coordinates and the target area, perform plane fitting according to the corner point depth value of the target corner point, and obtain a target depth map corresponding to each plane in a plurality of fitting planes;

[0014] A depth map intersection algorithm is used to perform depth map intersection calculation on the target depth map to obtain a layout depth map of the target indoor scene image.

[0015] According to the above aspects and any possible implementation, an implementation is further provided.

[0016] Corner points include: layout corner points and image corner points;

[0017] The layout corner points include layout boundary points and layout interior points; the layout boundary points are the intersection points of two planes and the image boundary; the layout interior points are the intersection points of three planes.

[0018] According to the above aspects and any possible implementation, an implementation is further provided.

[0019] The deep learning model is obtained by the following steps:

[0020] Obtaining a set of manually annotated 3D layout images; wherein the 3D layout images are annotated with corner pixel coordinates and corresponding depth values;

[0021] Inputting the 3D layout image set into a pre-built network model for training;

[0022] When the accuracy of the network model meets the preset threshold, the training is stopped to obtain a deep learning model;

[0023] Among them, the network model is constructed with three network branches.

[0024] According to the above aspects and any possible implementation, an implementation is further provided.

[0025] The training of the network model is supervised using a binary cross entropy loss function.

[0026] According to the above aspects and any possible implementation, an implementation is further provided.

[0027] The generating corner point coordinates according to the corner point heat map includes:

[0028] Using the non-maximum suppression algorithm, the pixel coordinates of the corner points are extracted from the Gaussian heat map of the predicted corner points as the corner point coordinates.

[0029] According to the above aspects and any possible implementation, an implementation is further provided.

[0030] The clustering of the embedding vectors to obtain a rough segmentation map, and performing dilation and corrosion operations on each plane area of ​​the rough segmentation map to obtain a target area, including:

[0031] Applying a mean shift clustering algorithm to the embedding vector to generate a rough segmentation map;

[0032] Perform dilation and erosion operations on each plane region of the coarse segmentation map;

[0033] The expanded image and the eroded image are subjected to a bitwise XOR operation to obtain the middle area formed by the expanded boundary and the eroded boundary as the target area.

[0034] According to the above aspects and any possible implementation, an implementation is further provided.

[0035] The determining of the target corner point according to the corner point coordinates and the target area, performing plane fitting according to the corner point depth value of the target corner point, and obtaining a target depth map corresponding to each plane in a plurality of fitting planes includes:

[0036] The corner points falling in the middle area are taken as target corner points, and the least square method is used to calculate the corresponding corner point depth values ​​to obtain the plane depth parameters;

[0037] According to the plane depth parameters, target depth maps of multiple planes are obtained by fitting.

[0038] According to a second aspect of the present disclosure, a device for estimating 3D layout of an indoor scene based on corner depth prediction is provided. The device comprises:

[0039] A data acquisition module, used to acquire a target indoor scene image;

[0040] A model output module, used to input the target indoor scene image into a deep learning model, and output a predicted corner point heat map, corner point depth value and embedding vector;

[0041] A coordinate determination module, used for generating corner point coordinates according to the corner point heat map;

[0042] A rough segmentation module is used to cluster the embedding vectors to obtain a rough segmentation map, and perform dilation and erosion operations on each plane area of ​​the rough segmentation map to obtain a target area;

[0043] A depth map generation module is used to determine a target corner point according to the corner point coordinates and the target area, perform plane fitting according to the corner point depth value of the target corner point, and obtain a target depth map corresponding to each plane in a plurality of fitting planes;

[0044] The depth map generation module is further used to perform depth map intersection calculation on the target depth map using a depth map intersection algorithm to obtain a layout depth map of the target indoor scene image.

[0045] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0046] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0047] The embodiments of the present disclosure provide a method and device for estimating the 3D layout of an indoor scene based on corner point depth prediction, which dilates and erodes semantic labels to obtain coplanar corner points, thereby obtaining the plane depth parameters of each plane and restoring the indoor layout.

[0048] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0050] Figure 1 A flowchart of a method for estimating 3D layout of an indoor scene based on corner depth prediction according to an embodiment of the present disclosure is shown;

[0051] Figure 2 A block diagram of an indoor scene 3D layout estimation device based on corner point depth prediction according to an embodiment of the present disclosure is shown;

[0052] Figure 3 A schematic block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown;

[0053] Figure 4 An example diagram of the shortest distance from a real corner point to a fitting plane according to an embodiment of the present disclosure is shown;

[0054] Figure 5 A schematic diagram of the overall process of restoring indoor layout by predicting corner points with depth according to an embodiment of the present disclosure is shown;

[0055] Figure 6 An exemplary schematic diagram showing different types of corner points with the shortest distances to adjacent planes according to an embodiment of the present disclosure;

[0056] Figure 7 A schematic diagram of qualitative result comparison of the Matterport 3D-Layout dataset according to an embodiment of the present disclosure is shown; wherein (a) shows an input image, (b) shows predicted corner points with depth, (c) shows a segmentation mask obtained by clustering two predicted embedding vectors, (d) shows an estimated layout depth map, (e), (f), (g) and (h) respectively show the 2D layout of the present disclosure and the 2D layouts of other methods;

[0057] Figure 8 A schematic diagram of qualitative result comparison of the LSUN dataset according to an embodiment of the present disclosure is shown; wherein (a) shows an input image, (b) shows predicted corner points with depth, (c) shows a segmentation mask obtained by clustering two predicted embedding vectors, (d) shows an estimated layout depth map, (e) and (f) respectively show the 2D layout of the present disclosure and the 2D layout of the corresponding other methods;

[0058] Fig. 9A schematic diagram of test results after different cropping of the same image according to an embodiment of the present disclosure is shown; wherein (a) is an image after five different croppings of the original image, (b) is a test result of the GeoLayout method, and (c) is a test result of the method of the present disclosure. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0060] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0061] In this disclosure, Figure 5 The overall process diagram of predicting corner points with depth to restore the indoor layout is shown in the figure. The position information and depth information of the layout corner points and image corner points in the predicted image are proposed. At the same time, two additional embedding vectors are predicted, and a rough layout segmentation map is obtained by clustering them. The predicted points are classified by dilation and erosion operations to obtain the plane depth parameters of each plane, and then the indoor layout is restored.

[0062] First, a method to recover indoor layout by learning corner points with depth does not rely on location context information. A method based on planar region dilation, erosion and XOR is proposed to find coplanar corner points.

[0063] Second, a geometric constraint of point-to-plane distance is proposed, which can effectively enhance the learning of corner depth.

[0064] Third, experiments are conducted on the Matterport3D-Layout dataset and two 2D benchmark datasets to verify the effectiveness and robustness of the method.

[0065] Figure 1 A flowchart of a method 100 for estimating 3D layout of an indoor scene based on corner depth prediction according to an embodiment of the present disclosure is shown. The method 100 includes:

[0066] Step 110: Acquire a target indoor scene image.

[0067] In some embodiments, an indoor scene image for which 3D layout estimation of the indoor scene is required is acquired.

[0068] In some embodiments, in indoor scenes, layout corners (intersection points between walls) are significant feature points in the indoor layout, providing important information for the indoor layout estimation task. Figure 6 The following is a schematic diagram showing different types of corner points with the shortest distance to adjacent planes, where corner points are divided into three categories, including layout inliers (intersections of three planes), layout boundary points (intersections of two planes and the image boundary), and four image corner points. For example, a layout boundary point is the intersection of two walls, so its distance to walls labeled 1 and 2 is the shortest. A layout inlier is the intersection of three walls, so its distance to walls labeled 1, 3, and 4 is the shortest. An image corner is the intersection of a wall and the image boundary, so its distance to the wall labeled 4 is the shortest. According to this definition, it is ensured that at least three points with depth can be found in any plane, so that its plane depth parameters can be fitted.

[0069] Step 120: input the target indoor scene image into a deep learning model, and output a predicted corner point heat map, corner point depth value, and embedding vector.

[0070] In some embodiments, the deep learning model is obtained by the following steps: obtaining a set of manually annotated 3D layout images; wherein the 3D layout images are annotated with corner pixel coordinates and corresponding depth values; inputting the 3D layout image set into a pre-built network model for training; when the accuracy of the network model meets the preset threshold, the training is stopped to obtain a deep learning model; wherein the network model uses swin-transformer as an encoder, neural window FC-CRFs module as a decoder, and a pyramid pooling module (PPM) is used to summarize the information of the entire image. Finally, a predicted depth map is obtained by a convolution and upsampling operation. Among them, the output part of the network model is divided into 3 output branches, i.e., three network branches. The first branch is used to predict the position of the corner point; the second branch is responsible for predicting the depth of the corner point; and the third branch is used to predict the embedding vector. Considering that the positions of the four image corner points are fixed, only the positions of the layout corner points need to be learned.

[0071] The number of output channels of the first branch is adjusted to 2, and Gaussian heat maps representing the layout boundary points and layout interior points in the image are output respectively. Gaussian blur is added to the image containing only the real layout boundary points and layout interior points, and the binary cross entropy loss function is used. Unsupervised training. Since the number of positive and negative samples is too different, a larger loss weight is given to the positive sample. The specific formula is as follows:

[0072]

[0073] in, and They represent the predicted position heat map. The probability that each pixel is a layout boundary point or a layout interior point; and The actual layout boundary point and the layout internal point position map pixels; and are the weights of whether the i-th pixel is a layout boundary point or a layout internal point; N is the total number of pixels in the predicted position heat map; is the loss of the layout boundary points, is the loss of the points inside the layout.

[0074] The second output branch has an output channel number of 1, which is used to predict the depth value of the corner point. Loss weight is only assigned to the corner point position after Gaussian blurring, and L1loss is used to supervise the learning of the depth value at the corner point. The specific loss function is as follows:

[0075]

[0076] Among them, Lz is the corner loss, N is the total number of corners, Pi is the corner probability, , E represents the position map of the four image corner points, that is, the pixel values ​​at the four image corner points are 1, and the pixel values ​​at the other points are 0. is the predicted depth value at pixel i, Represents the true depth value at pixel i.

[0077] In addition, the scale-invariant logarithmic SILog loss is used as a supplement to guide the model to learn the depth value at the corner point. It should be noted that the loss weight is not directly given to the depth value of the entire image, but only the loss weight is given to the layout corner point and the image corner point position and its surrounding points. First, calculate the logarithmic difference between the depth map of the predicted point and the depth of the real point:

[0078]

[0079] in, represents a matrix composed of 0 and 1, and is the value of the element in the matrix. When the pixel value of the i-th pixel is 1, it means that the logarithmic difference of the depth value at the pixel is calculated. Then for the K pixels with valid depth values ​​in the image, the scale invariant loss is calculated as:

[0080]

[0081] Among them, Ls is the scale invariant loss, is the variance minimization factor, Set the scale constant to 10, Set to 0.85, K is the number of pixels with valid depth values.

[0082] In addition, in general, the distance from the layout boundary point to its two adjacent faces is the shortest and is 0, the distance from the layout internal point to its three adjacent faces is the shortest and is 0, and the distance from the image corner point to its one adjacent face is the shortest and is 0. Therefore, a geometric constraint of point-to-face distance is proposed , to supervise the learning of corner depth values, the specific formula is as follows:

[0083]

[0084] in , , is the distance from the intersection of the three faces to each face, o is the number of the intersection, i is one of the three intersections selected. is the distance from the intersection of the two faces and the image boundary to each face, r is the sequence number of the intersection, j is one of the first two intersections selected, is the distance from the image corner point to each face, e is the serial number of the corner point, n is the number of real three-face intersection points, and m is the number of real two-face intersection points with the image boundary.

[0085] For example, Figure 4 The following is an example of the shortest distance from the real corner point to the fitting plane. Different types of corner points find the nearest fitting plane in the (u, v, 1 / z) coordinate system.

[0086] Therefore, the total depth constraint is:

[0087]

[0088] The third branch is used to learn two embedding vectors, so the number of output channels is set to 2. Map all pixels to the embedding space. The embeddings of pixels belonging to the same plane should be as close as possible, and the embeddings of pixels in different planes should be as far away as possible. And use the discriminant loss , which penalizes pixel embeddings that are far away from each other on the same face and pixel embeddings that are close to each other on different faces:

[0089]

[0090] Where C is the number of planes in the true value, n c is the number of pixels in plane c, X i is the pixel embedding, X cis the average embedding of plane C, δv and δd are the thresholds of pull loss and push loss respectively, α is the weight of pull loss, β is the weight of push loss, A and B represent the corresponding output channels, X cA is the average embedding of plane C output by A output channels, X cB is the average embedding of plane C output by A output channels.

[0091] Finally, the total training loss is:

[0092]

[0093] Step 130: Generate corner point coordinates according to the corner point heat map.

[0094] In some embodiments, step 130 includes: using a non-maximum suppression algorithm to extract the pixel coordinates of the corner points from the Gaussian heat map of the predicted corner points as the corner point coordinates.

[0095] In some embodiments, the non-maximum suppression algorithm NMS is used to extract specific pixel coordinates from the Gaussian heat map of the predicted layout boundary points and the Gaussian heat map of the layout interior points, and then the depth value corresponding to the coordinate is found in the predicted depth map. At this point, we have obtained the predicted corner points with depth.

[0096] Step 140: cluster the embedding vectors to obtain a rough segmentation map, and perform dilation and erosion operations on each plane area of ​​the rough segmentation map to obtain a target area.

[0097] In some embodiments, step 140 includes: applying a mean shift clustering algorithm to the embedding vector to generate a rough segmentation map; performing dilation and erosion operations on each plane region of the rough segmentation map; performing a bitwise XOR operation on the dilated image and the eroded image to obtain an intermediate region formed by the dilation boundary and the erosion boundary as the target region. For example, applying a mean shift clustering algorithm to the two predicted embedding vectors to generate a rough layout segmentation result. Subsequently, each plane is taken out in turn according to the rough layout segmentation map, and the plane is dilated and eroded using a structural element with a square shape and a size of 10.

[0098] Step 150: determining a target corner point according to the corner point coordinates and the target area, performing plane fitting according to the corner point depth value of the target corner point, and obtaining a target depth map corresponding to each plane in a plurality of fitting planes.

[0099] In some embodiments, step 150 includes: taking the corner points falling in the middle area as points on the corresponding fitting plane, and fitting the target depth map of the corresponding plane according to the corresponding depth values. That is, when the predicted boundary points, internal points and four image corner points fall in the middle area, they are classified as points on the plane, that is, target corner points. Then, the pixel coordinates of the three corner points on the same plane are fitted with the depth values ​​of the corresponding corner points to obtain all planes. That is, the pixel coordinates of the corner points and the corresponding predicted depth values ​​are fitted using the least squares method to obtain the fitted plane. Among them, a point on a 3D plane can be projected onto the image plane and satisfies the following equation:

[0100]

[0101] Using the above equation, the points classified as being on the same plane are fitted with the least squares method to obtain the parameters of each surface. , (u, v, 1 / Z) represents the coordinate system.

[0102] Step 160: Perform depth map intersection calculation on the target depth map using a depth map intersection algorithm to obtain a layout depth map of the target indoor scene image.

[0103] In some embodiments, the depth map of each surface obtained in step 150 is used to generate a layout through a depth map intersection algorithm, that is, a planar layout is obtained by intersecting planar depth maps with each other.

[0104] In some embodiments, based on Hedau and LSUN, two commonly used benchmark datasets in layout estimation tasks, they both rely on real two-dimensional segmentation to describe the scene layout and lack real depth information. Then, in order to apply the above method on these datasets, first pre-training is performed on the Matterport3D-Layout dataset, and then the pre-trained model is fine-tuned on the benchmark dataset. Then, the above network structure is used to learn corner points with depth and two embedding vectors. Using L cor To constrain the predicted corner position, and L embed The embedding vector for the constraint prediction.

[0105] At the same time, the layout types in the Hedau and LSUN datasets are both cuboids, so the 3D layout of the cube scene can be obtained by finding the minimum depth value of all main plane depth maps:

[0106]

[0107] Among them, Zi is the layout depth value, f is the plane number, and i is the pixel point in the corresponding plane.

[0108] Since LSUN provides the true layout segmentation and 2D coordinates of the corner points, first classify the layout corner points and the four image corner points into their respective planes. Next, perform least squares fitting on the true pixel coordinates of all points on the same plane and the predicted depth values ​​corresponding to the points to obtain the plane parameters. Then, calculate the depth map and extract the minimum value at each pixel. The generated depth map is used as a pseudo-label to supervise the learning of the depth values ​​at the layout corner points and the four image corner points. The loss function of the predicted depth map is defined as follows:

[0109] In addition, the geometric constraint L of the point-to-surface distance mentioned above is added P-P .

[0110]

[0111] In addition, since the true depth value of the layout plane is generally within a certain range, the following loss is given to supervise the learning to prevent the depth value from being too high or too low:

[0112]

[0113] in, is the lower bound of the depth value, The upper bound of the depth value.

[0114] However, if we only rely on the above two losses for supervised training, the learning model will tend to make each plane produce a closer depth value in order to minimize the loss. Therefore, the use of cross entropy loss makes the depth value under each label have a larger gap.

[0115]

[0116] in is the scaling factor, F is the number of planes, It is the i-th pixel of the depth map spliced ​​according to the 2D real segmentation map. So the total loss is:

[0117]

[0118] In this way, the Matterport3D-Layout, LSUN and Hedau datasets can be used for training and testing in the above manner.

[0119] Below, the above process of this application is described in detail with the three datasets of Matterport3D-Layout, LSUN and Hedau:

[0120] First, the size of the input RGB images of indoor scenes is adjusted to 224×224 using bilinear interpolation, and the output sizes of the three output branches of the network are 112×112×2, 112×112×1, and 112×112×2 respectively. The training data is enhanced using data augmentation methods such as color lighting and color jitter. The network was trained for 200 epochs using the pytorch environment on a server with an Nvidia RTX4090 GPU. Adam was used as the optimizer, the batch size was set to 16, the initial learning rate was 10-4, and the weight decay was 10-4. During training, the learning rate was reduced by 30% every 50 rounds of training. The hyperparameters in the experiment were set as follows: , , .

[0121] About the dataset: The Matterport3D-Layout dataset is obtained by combining manual annotation and plane fitting algorithms. It is specifically used for 3D layout estimation research and provides real 3D layout information. In addition, the pixel coordinates of the layout corners and the depth values ​​corresponding to these corners are provided. Therefore, it is very suitable for learning the corner position and its corresponding depth information. The dataset consists of indoor images from 90 different buildings, showing good layout diversity. The training set includes 64 buildings, with a total of 4939 images; the validation set includes 6 buildings, with a total of 456 images. The test set contains the remaining 20 buildings, with a total of 1965 images.

[0122] The LSUN dataset contains 4000 training images, 394 validation images, and 1000 test images, including the true 2D layout segmentation masks and the pixel coordinates of the layout corners.

[0123] The Hedau dataset consists of 209 training images and 105 testing images.

[0124] Regarding evaluation indicators: The evaluation criteria used for the Matterport3D-Layout dataset are concentrated on the layout depth map. The evaluation criteria used include mean square error (rms), mean relative error (rel), mean log10 error (log10), threshold accuracy , 3D corner error (represents the Euclidean distance between the 3D layout corners in the camera coordinate system and the ground truth). For the LSUN and Hedau datasets, the metrics used to evaluate the proposed model include pixel segmentation error And the corner position error Among them, the pixel error refers to the accuracy of each pixel surface label between the true and estimated ones; the corner error is the Euclidean distance between the true corner point and the estimated corner point normalized by the length of the image diagonal.

[0125] 1) Results on the Matterport3D-Layout dataset:

[0126] We first train the network on the Matterport3D-Layout training set, using the real layout segmentation map and the real corner points with depth as supervision signals. We evaluate our method on the test set and compare it with other state-of-the-art methods.

[0127] Quantitative results: The following table lists the quantitative comparison results. It can be observed that this method shows the best performance in both 2D layout estimation indicators and 3D layout estimation indicators:

[0128]

[0129] At the same time, in order to reveal the new geometric constraint of the point-to-plane distance (L P-P ) and conducted an ablation experiment on the Matterport3D-Layout dataset, as shown in the following table:

[0130]

[0131] That is, compared with the addition of L P-P The results before and after show that adding L P-P The model after (GeoLayout+L P-P ) has been significantly improved in most indicators, especially in 2D indicators. P-P )” indicates that L is not added P-P The model, “Ours(w / L P-P )” means adding L P-P model. It can be found that “Ours(w / L P-P )” performed best.

[0132] Qualitative results: Figure 7The schematic diagram of qualitative result comparison of the Matterport3D-Layout dataset is shown, in which the selected room types include three rectangular rooms and three non-rectangular rooms. The input image is shown in (a), and the predicted corner points with depth are shown in (b). The segmentation mask obtained by clustering the two predicted embedding vectors is shown in (c). The estimated layout depth map is shown in (d). (e), (f), (g) and (h) respectively show the 2D layout of the present disclosure and the 2D layout of the corresponding other methods. It can be seen that the present disclosure produces good layout estimation results both in rectangular rooms and non-rectangular rooms. Experimental results show that this method can obtain more accurate three-dimensional layout results.

[0133] 2) Results on the LSUN dataset and the Hedau dataset:

[0134] In order to evaluate the generalization ability of the disclosed model, the model trained on the Matterport 3D-Layout dataset is directly evaluated on the LSUN validation set and the Hedau test set. This process does not require additional training. The following table w / o Fine-tune represents the results of direct testing:

[0135]

[0136] From the results, it can be seen that even without additional training, the disclosed model still produces relatively reliable results and has good generalization. Next, it was fine-tuned on the LSUN dataset, divided into 2D layout estimation and 3D layout estimation. It can also be found that although the LSUN dataset does not contain the real layout depth map, the performance has been significantly improved, and compared with other methods, better performance is still achieved on the LSUN dataset. Finally, the model fine-tuned on the LSUN dataset was tested on the Hedau test set, and the results were also better than other methods in the table, which also proved the effectiveness of this method.

[0137] Qualitative results: Figure 8 The schematic diagram of qualitative result comparison of the LSUN dataset is shown, where the selected room types include three rectangular rooms and three non-rectangular rooms. The input image is shown in (a), and the predicted corner points with depth are shown in (b). The segmentation mask obtained by clustering the two predicted embedding vectors is shown in (c). The estimated layout depth map is shown in (d). (e) and (f) respectively show the 2D layout of the present disclosure and the 2D layout of the corresponding other methods. It can be seen that although the LSUN dataset does not contain the real layout depth map, the performance is still significantly improved.

[0138] 3) Stability test:

[0139] In order to test the independence of the method disclosed in this paper on the position in the image, the following experiment was conducted. Each image in the Matterport3D-Layout test set was cropped five times in different ways, and the method disclosed in this paper and the GeoLayout method were used to test them respectively. The standard deviation of the error of each image was calculated, that is, Fig. 9 The following is a schematic diagram of the test results of the same image after different cropping. There are two groups of images in total, where (a) is the image after five different croppings of the original image, (b) is the test result of the method disclosed in this disclosure, and (c) is the test result of the GeoLayout method. Since the true corner coordinates of the cropped image are unknown, the 2D index only considers pixel errors. The results are shown in the following table:

[0140]

[0141] It can be found that the standard deviation of each error of the method disclosed in the present invention is smaller than that of GeoLayout, indicating that the method disclosed in the present invention has stronger stability. When the original image is used for testing, both the method disclosed in the present invention and GeoLayout can obtain a better 2D layout. However, after cropping, GeoLayout has obvious prediction errors, and the results produced by the method disclosed in the present invention after different cropping are closer to the true value. The experimental results show that compared with the method of directly learning the plane depth parameters, the model disclosed in the present invention is more stable and has better consistency.

[0142] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0143] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0144] Figure 2 FIG. 2 shows a block diagram of an indoor scene 3D layout estimation device 200 based on corner depth prediction according to an embodiment of the present disclosure. Figure 2 As shown, the device 200 includes:

[0145] A data acquisition module 210 is used to acquire a target indoor scene image;

[0146] A model output module 220, configured to input the target indoor scene image into a deep learning model and output a predicted corner point heat map, corner point depth value and embedding vector;

[0147] A coordinate determination module 230, configured to generate corner point coordinates according to the corner point heat map;

[0148] A rough segmentation module 240 is used to cluster the embedding vectors to obtain a rough segmentation map, and perform dilation and erosion operations on each plane area of ​​the rough segmentation map to obtain a target area;

[0149] A depth map generating module 250 is used to determine a target corner point according to the corner point coordinates and the target area, perform plane fitting according to the corner point depth value of the target corner point, and obtain a target depth map corresponding to each plane in a plurality of fitting planes;

[0150] The depth map generation module 250 is further configured to perform depth map intersection calculation on the target depth map using a depth map intersection algorithm to obtain a layout depth map of the target indoor scene image.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0152] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0153] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0154] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0155] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0157] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0162] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0163] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0164] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for estimating 3D layout of indoor scenes based on corner depth prediction, characterized in that: include: Acquire the target indoor scene image; Input the target indoor scene image into a deep learning model, and output a predicted corner point heat map, corner point depth value, and embedding vector; Generate corner point coordinates according to the corner point heat map; Clustering the embedding vectors to obtain a rough segmentation map, and performing dilation and erosion operations on each plane region of the rough segmentation map to obtain a target region; Determine a target corner point according to the corner point coordinates and the target area, perform plane fitting according to the corner point depth value of the target corner point, and obtain a target depth map corresponding to each plane in a plurality of fitting planes; Performing depth map intersection calculation on the target depth map using a depth map intersection algorithm to obtain a layout depth map of the target indoor scene image; The deep learning model is obtained by the following steps: Obtaining a set of manually annotated 3D layout images; wherein the 3D layout images are annotated with corner pixel coordinates and corresponding depth values; Inputting the 3D layout image set into a pre-built network model for training; When the accuracy of the network model meets the preset threshold, the training is stopped to obtain a deep learning model; The network model is constructed with three network branches: The first branch is used to predict the position of the corner points; the second branch is responsible for predicting the depth of the corner points; the third branch is used to predict the embedding vector; The number of output channels of the first branch is adjusted to 2, and Gaussian heat maps representing the layout boundary points and layout interior points in the image are output respectively. Gaussian blur is added to the image containing only the real layout boundary points and layout interior points, and the binary cross entropy loss function L is used. cor supervised training; The loss weight at the positive sample is as follows: L cor =L B +L I Among them, B i and I i They represent the probability that the i-th pixel in the predicted position heat map is a layout boundary point and a layout interior point respectively; B * with I * They respectively represent the position map of the actual layout boundary points and the position map of the layout interior points after adding Gaussian blur; and are the weights of whether the i-th pixel is a layout boundary point or a layout interior point; The number of output channels of the second output branch is 1; loss weights are assigned to the corner points after Gaussian blurring, and L1loss is used to supervise the learning of depth values ​​at the corner points; the specific loss function is as follows: Where P = B * +I * +E, E represents the position map of the four image corner points, that is, the pixel values ​​at the four image corner points are 1, and the pixel values ​​at the other points are 0; is the predicted depth value at pixel i, Represents the true depth value at pixel i; The scale-invariant logarithmic SILog loss is used as a supplement; The loss weights of the layout corner points, image corner points and their surrounding points: First, calculate the logarithmic difference between the depth map of the predicted point and the depth of the actual point: Among them, w s represents a matrix composed of 0 and 1. When the pixel value of the i-th pixel is 1, it means that the logarithmic difference calculation is performed on the depth value at the pixel. Then for the K pixels with valid depth values ​​in the image, the scale-invariant loss is calculated as: Where λ is the variance minimization factor, the scale constant η is set to 10 and λ is set to 0.85; The distance from the layout boundary point to its two adjacent faces is the shortest and is 0, the distance from the layout interior point to its three adjacent faces is the shortest and is 0, and the distance from the image corner point to its one adjacent face is the shortest and is 0; Geometric constraint L of point-to-plane distance P-P , the specific formula is as follows: where d o，i ≤d o，i+1 ,h r，j ≤h r，j+1 , d o,j is the distance from the intersection of the three faces to each face, o is the number of the intersection, i is one of the three selected intersections, and h is r,j is the distance from the intersection of the two faces and the image boundary to each face, r is the number of the intersection, j is one of the first two intersections selected, S e is the distance from the image corner to each face, e is the serial number of the corner, n is the number of real three-face intersections, and m is the number of real two-face intersections with the image boundary; The total depth constraint is: L dep =L S +L Z +L P-P ; The third branch is used to learn two embedding vectors, so the number of output channels is set to 2; Map all pixels to the embedding space; And use the discriminative loss L embed : L embed =αL pull βL push Where C is the number of planes in the true value, n c is the number of pixels in plane c, X i is the pixel embedding, X c is the average embedding of plane C, δv and δd are the thresholds of pull loss and push loss respectively, α is the weight of pull loss, and β is the weight of push loss; The total training loss is: L 3D-all =L cor +L dep +L embed .

2. The method according to claim 1, characterized in that: Corner points include: layout corner points and image corner points; The layout corner points include layout boundary points and layout interior points; the layout boundary points are the intersection points of two planes and the image boundary; the layout interior points are the intersection points of three planes.

3. The method according to claim 1, characterized in that The training of the network model is supervised using a binary cross entropy loss function.

4. The method according to claim 1, characterized in that: The generating corner point coordinates according to the corner point heat map includes: Using the non-maximum suppression algorithm, the pixel coordinates of the corner points are extracted from the Gaussian heat map of the predicted corner points as the corner point coordinates.

5. The method according to claim 1, characterized in that The clustering of the embedding vectors to obtain a rough segmentation map, and performing dilation and corrosion operations on each plane area of ​​the rough segmentation map to obtain a target area, including: Applying a mean shift clustering algorithm to the embedding vector to generate a rough segmentation map; Perform dilation and erosion operations on each plane region of the coarse segmentation map; The expanded image and the eroded image are subjected to a bitwise XOR operation to obtain the middle area formed by the expanded boundary and the eroded boundary as the target area.

6. The method according to claim 5, characterized in that The determining of the target corner point according to the corner point coordinates and the target area, performing plane fitting according to the corner point depth value of the target corner point, and obtaining a target depth map corresponding to each plane in a plurality of fitting planes includes: The corner points falling in the middle area are taken as target corner points, and the least square method is used to calculate the corresponding corner point depth values ​​to obtain the plane depth parameters; According to the plane depth parameters, target depth maps of multiple planes are obtained by fitting.

7. The device of the method for estimating 3D layout of indoor scenes based on corner depth prediction according to claim 1, characterized in that: include: A data acquisition module, used to acquire a target indoor scene image; A model output module, used to input the target indoor scene image into a deep learning model, and output a predicted corner point heat map, corner point depth value and embedding vector; A coordinate determination module, used for generating corner point coordinates according to the corner point heat map; A rough segmentation module is used to cluster the embedding vectors to obtain a rough segmentation map, and perform dilation and erosion operations on each plane area of ​​the rough segmentation map to obtain a target area; A depth map generation module is used to determine a target corner point according to the corner point coordinates and the target area, perform plane fitting according to the corner point depth value of the target corner point, and obtain a target depth map corresponding to each plane in a plurality of fitting planes; The depth map generation module is further used to perform depth map intersection calculation on the target depth map using a depth map intersection algorithm to obtain a layout depth map of the target indoor scene image.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images