A perspective image C2P conversion and layout generation method for room layout estimation

Abstract

The present invention discloses a perspective image C2P conversion and layout generation method for room layout estimation. It includes: acquiring a target indoor scene image; inputting the target indoor scene image into a deep learning model, outputting a predicted complete depth map, a global plane visibility confidence map, a wall category, and a plane semantic label; classifying the planes, defining anchor points evenly distributed on the perspective image, and for the dominant plane, calculating the centroid coordinates of the corresponding segmentation area, and assigning the label to the plane corresponding to the index of the anchor point closest to the centroid to obtain a rough layout segmentation mask. Generate plane depth parameters from the complete depth map and the plane visibility confidence. Generate a plane depth map using C2P conversion technology, and accurately estimate the plane layout of the room by integrating sequential plane depth maps. The present invention enables C2P transformation to be flexibly applied to perspective images, providing strong technical support for plane depth reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a perspective view C2P conversion and layout generation method for room layout estimation. Background Art

[0002] Room layout estimation is a challenging problem in computer vision, which aims to infer the overall spatial layout of an indoor environment from perspective images. Applications include scene reconstruction, indoor localization, and augmented reality.

[0003] Room layout is determined by major indoor planes, specifically the floor, ceiling, and walls. Recovering 3D room layout essentially involves reconstructing these major indoor planes, which is a fundamental challenge in layout estimation. Some recent studies have reconstructed these major planes in perspective images by learning plane parameters. However, a significant limitation of these methods is the lack of a mechanism to guide the network to focus on the visible regions of major indoor planes, which makes the prediction susceptible to occlusions. In addition, the determination of plane parameters is influenced not only by visual appearance but also by the positional context of the planes in the image. This dual dependency increases complexity and places an additional burden on network fitting. Summary of the Invention

[0004] The object of the present invention is to provide a perspective view C2P conversion and layout generation method for room layout estimation, so as to solve the technical problems existing in the prior art.

[0005] To achieve the above-mentioned object, the perspective view C2P conversion and layout generation method for room layout estimation of the present invention includes:

[0006] Step 1. Use a parametric method to represent the depth of a plane in a perspective image. According to the pinhole camera model, the projection of a 3D point (x, y, z) from the camera coordinate system to a pixel (u, v) in the perspective image is as follows:

[0007]

[0008] where f x , f y is the focal length, (u0, v0) is the principal point, if the equation is If any plane is projected onto the image, its pixel coordinates and depth values ​​will satisfy This equation can be restated as:

[0009]

[0010] Here, It is defined as the planar depth parameter of a plane in a perspective image;

[0011] Step 2: Based on the plane depth parameterization, C2P conversion is introduced;

[0012] Step 3: Design of perspective images; classify planes and define M×M anchor points evenly distributed on the perspective image. For each dominant plane including the ceiling and floor, calculate the centroid coordinates of the corresponding segmented region. The label assigned to the plane corresponds to the index of the anchor point closest to the centroid.

[0013] Step 4: Sequential planar depth map integration.

[0014] Furthermore, the step 2 introduces C2P conversion including: the goal of C2P conversion is to derive k The plane depth parameter of plane k is used to facilitate plane depth reconstruction; let plane be k, let Z represent the complete depth map of the perspective image, Represents the plane visibility confidence, Q k Denotes the target plane depth parameter of plane k, constructed using the homogeneous pixel coordinates obtained from Z and their corresponding inverse depth values. and The formula is as follows:

[0015]

[0016] To ensure consistency, d i To represent the inverse depth value of the i-th pixel in the perspective image, that is Second, use the elements on the diagonal Construct a diagonal matrix Calculate the target plane depth parameter Q k as follows:

[0017]

[0018] For planar depth map reconstruction, construct and in

[0019] Given The reconstructed depth map of plane k is calculated as follows:

[0020]

[0021] The goal of C2P transformation is to derive a k The C2P transformation is differentiable for perspective images, allowing the gradient of the loss function applied to the estimated plane depth parameters and the reconstructed plane depth map to be back-propagated to the plane visibility confidence.

[0022] Furthermore, the plane visibility confidence is learned autonomously without direct supervision, i.e., forcing the network to fit all visible plane regions is unnecessary; instead, allowing the network to independently identify and focus on the most credible visible plane regions leads to more favorable results.

[0023] Furthermore, the step 3 further includes: obtaining a plane k whose region centroid is closest to the anchor point in the i-th row and j-th column, where k = (i-1)M+j. Through this labeling strategy, a total of M 2 different plane categories; the output branch responsible for predicting the plane category produces a probability map aligned with the distribution space of the anchors If there is a plane k in the image, then The i-th row and j-th column of The value in should be 1, otherwise it should be 0; The learning of is supervised by the binary map generated by the ground truth layout segmentation mask, denoted as Taking into account There is a significant imbalance between positive and negative samples, using and The weighted cross entropy loss between is used as the loss function:

[0024]

[0025] where t is the weighting factor which is experimentally set to t=25.

[0026] Furthermore, the step 3 further includes: using a decoupling method, the number of output channels is increased from M to 2 is reduced to 2M, where M channels are dedicated to predicting the x-axis component α∈R h×w×M , the remaining M channels are dedicated to predicting the y-axis component β∈R h×w×M , the j-th channel mapping of α is represented as α (j) (j=1, 2, ···, M), which is a probability map indicating whether the plane category of each pixel is associated with an anchor in the j-th column; similarly, β (i) (i=1, 2, ···, M) indicates whether the plane category of each pixel is associated with one of the anchors in the i-th row, so α j and β i The joint probability represents the semantic label of the plane associated with the anchor in the i-th row and j-th column, and the plane category is (i-1)M+j, defined as To represent M 2 All semantic labels of the plane category, since α and β are independent, Each channel map in can be obtained by element-wise multiplication of the channel maps in α and β:

[0027]

[0028] in Indicates the right-to-speak operation; Based on the above analysis, the corresponding output branch is used to predict the decoupled plane semantic labels of the 2M output channels. The training objective of the decoupled planar semantic labels represented by is composed of the sum of the binary cross entropy loss and the dice loss specified in the formula.

[0029] Furthermore, the step 4 further comprises:

[0030] Step 4.1: Determine the wall categories in the image and generate planar depth maps of the ceiling, floor, and each wall using C2P conversion technology.

[0031] Step 4.2, distinguishing the selected plane into walls and non-walls by classifying the estimated plane depth parameters;

[0032] Step 4.3: Through a detailed integration process, a complete and accurate 3D layout depth map is obtained.

[0033] Furthermore, the step 4.1 further includes: using the plane category prediction C, defining a threshold δ c , and select the predicted value greater than δ c Whenever the wall category is selected, its two adjacent categories are checked. If any of these adjacent categories is selected, the dice loss is calculated to evaluate the similarity between the predicted semantic labels of these adjacent planes. The dice loss exceeds a predefined threshold δ d Indicates that adjacent predictions correspond to different walls, prompting both to be retained. On the contrary, when the dice loss is lower than δ d , it prompts to keep only the wall category with higher prediction value.

[0034] Furthermore, the step 4.2 further comprises: having an estimated plane depth parameter The selected plane k, the second parameter σ k is proportional to the perpendicular component of the plane normal if the ratio Below a predefined threshold δ q , classify plane k as a wall, otherwise, classify plane k as a non-wall; after distinguishing all the walls, they are sorted from left to right based on the horizontal coordinates of the centroid associated with the plane semantic label, and the plane depth map is regarded as a 3D surface in a coordinate system defined by pixel coordinates and depth axis.

[0035] Furthermore, the step 4.2 further includes: analyzing the integral of the two planar depth maps as the intersection of the two surfaces in the 3D coordinate system, starting the process with the depth map of the leftmost wall, and gradually integrating the depth maps of subsequent walls. When a new wall is integrated, an intersection line is formed between the new planar depth map and the existing planar depth map; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label, trimming the area of ​​the planar depth map extending beyond the intersection line, thereby generating an updated layout depth map. This process will be repeatedly applied until the planar depth maps of all walls are integrated.

[0036] Furthermore, the perspective image network is built on the architecture designed for perspective depth estimation, which adopts SENET-154 as the encoder. The output branch for full depth map prediction includes a decoder, a multi-scale feature fusion module, and a refinement module.

[0037] The method of the present invention has the following advantages:

[0038] 1) This paper introduces the C2P transform, a differentiable operation specifically designed for robust planar depth reconstruction. This method, based on a derived parametric representation of planar depth, enables the C2P transform to be flexibly applied to perspective images, providing powerful technical support for planar depth reconstruction.

[0039] 2) This paper proposes an efficient and versatile learning framework for 3D layout estimation that is highly applicable to perspective images. Based on the network's predictions, we further propose an innovative sequential planar depth map integration method to generate accurate 3D layouts. This approach not only improves the accuracy and reliability of 3D layout estimation but also demonstrates the framework's potential for processing diverse image types.

[0040] 3) The proposed method achieves state-of-the-art performance in all benchmarks of layout estimation of perspective images.

[0041] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, which can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the C2P conversion graph;

[0043] Figure 2 Schematic diagram of C2P network for perspective image;

[0044] Figure 3Schematic diagram of eliminating duplicate wall category predictions based on the similarity of corresponding semantic labels. (a) Adjacent wall category predictions correspond to different walls. (b) Duplicate locations of adjacent wall category predictions, where only categories with higher activation values ​​are retained.

[0045] Figure 4 Schematic diagram of the visualization results of perspective layout estimation on the Matterport3D-Layout and InteriorNet-Layout datasets;

[0046] Figure 5 Schematic diagram of the qualitative comparison results of layout estimation and supervision plane visibility confidence;

[0047] Figure 6 Schematic diagram of the qualitative comparison results of learning the complete depth map and the layout depth map, evaluated on the Matterport3D-Layout dataset. The red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines. DETAILED DESCRIPTION

[0048] The technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments. However, it should be understood by those skilled in the art that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0049] This paper provides an innovative room layout estimation method, the core of which is plane depth reconstruction. First, a parametric expression of plane depth in perspective images is derived. Then, based on this foundation, a complete depth map to plane depth (C2P) conversion method is proposed, which can be effectively applied to perspective images. Figure 1As shown, through C2P transformation, the depth of the wall planes outlined in the image is reconstructed using the full depth map of the wall and the plane visibility confidence. By utilizing a parameterized representation of plane depth, the C2P transformation method transforms the plane depth reconstruction problem into the prediction of a combined depth map and plane visibility confidence. Specifically, the full depth map captures the depth information of visible plane regions and occluding objects, while the plane visibility confidence reflects the probability that the plane is observed at each pixel position, that is, not occluded by indoor objects or other salient planes. After obtaining the predicted full depth map and plane visibility confidence, a weighted least squares method is used to solve the problem. This solution is based on the premise that the reconstructed plane depth should be closely aligned with the full depth map in areas with high plane visibility confidence. This method can accurately estimate the room's floor plan. The detailed implementation steps and principles of C2P transformation of perspective images will be explained later.

[0050] Building on the C2P transformation approach, this paper further proposes an efficient learning framework for accurate 3D layout estimation. This framework employs two distinct architectures tailored for perspective images, aiming to simultaneously learn comprehensive depth maps and plane visibility confidences. Due to the differentiability of the C2P transformation, necessary constraints are imposed on both the predicted plane depth parameters and the reconstructed plane depth map, enabling the network to autonomously learn and optimize plane visibility confidences as needed. During the testing phase, an innovative sequential plane depth map integration method is introduced, which can be effectively applied to perspective images to generate high-quality 3D layouts. By learning a comprehensive depth map that includes both planes and objects, the network is able to fully understand the overall spatial structure of indoor scenes. Simultaneously, the joint learning of plane visibility confidences empowers the network to detect occlusions and focus on the visible regions of dominant planes. Compared to traditional methods that directly learn plane parameters, this approach demonstrates superior robustness in handling occlusions. Furthermore, the network's predictions are independent of specific locations, directly addressing the key issue of plane parameter learning. Although extending labeled 3D layout datasets is costly, our method is able to fully leverage abundant external RGB-D datasets to enhance its performance, further improving its practicality and generalization ability.

[0051] Room layout estimation based on perspective images:

[0052] Many traditional layout estimation methods adopt a classical framework, first generating a series of layout hypotheses based on detected vanishing points / lines or other image features. These hypotheses are then evaluated using a scoring function to select the optimal one. Some methods also leverage 3D priors and 3D reasoning for layout reasoning, such as classifying surface orientations based on vanishing lines and modeling the joint distribution of room layout and objects within the room. With the development of deep learning, many new methods have been introduced. One class of these methods utilizes deep networks to predict a 2D layout representation as a reference while simultaneously inferring a parameterized layout result. Many studies infer layouts by learning probabilistic maps representing the boundaries between dominant planes (i.e., ceiling, floor, and walls). Another class of methods involves learning semantic masks associated with dominant planes. Planar semantic masks provide richer layout information than boundaries, but learning can be affected by label ambiguity caused by wall definitions. Furthermore, some methods directly learn the locations of layout corners via one or more probabilistic maps. However, these networks face challenges in acquiring the underlying 3D knowledge about room layouts by learning only such 2D representations.

[0053] In recent years, three-dimensional representation and reasoning in layout estimation have attracted increasing attention. One method assumes that the room is a 3D template cube, thereby converting the layout estimation task into camera pose estimation. The study of Lin et al. adopts the 3D template cube assumption and uses the intermediate layout representation as a bridge to learn the camera pose, thereby facilitating the recovery of the 3D layout. Ren et al. use a reinforcement learning algorithm that gradually moves the camera inside the cube to recover the camera pose. Due to the assumption of the template cube, the generalization ability of this type of method is limited, especially for non-cubic rooms. Another type of method estimates the spatial configuration of the main indoor planes and reconstructs the 3D layout through the intersection of these planes. Zhang et al. proposed to learn the pixel-level plane depth parameters of the dominant planes and cluster these parameters to generate a plane depth map for 3D layout reconstruction. Stekovic et al. used PlaneRCNN to detect plane areas and plane parameters, and then inferred the 3D layout by solving a constrained discrete optimization problem. Yang et al. reconstructed 3D non-cubic layouts based on the detected plane bounding boxes and the jointly learned plane parameters. In addition, some studies focused on 3D reasoning, such as the orthogonality of adjacent planes. In addition, single-view Figure 3 The task of 3D plane reconstruction is to detect planar surfaces in indoor scenes, including dominant planes and indoor objects, and recover their planar parameters.

[0054] Plane depth reconstruction formula:

[0055] The present invention provides a C2P (complete depth map to plane depth) conversion method for perspective images. The core of this method is to generate plane depth parameters from the complete depth map and the plane visibility confidence. The plane visibility confidence here is a key indicator, which represents the probability that the plane is visible at each pixel position, that is, the probability that the position is not blocked by indoor objects or other major planes. In indoor scenes, areas with high visibility confidence are often parts where there are no obstacles and planar features can be clearly observed. Based on this understanding, it is expected that the plane depth map reconstructed using the estimated plane depth parameters can maintain a close alignment with the complete depth map of these areas. To achieve this goal, an optimization strategy is adopted, namely, determining the depth parameters of each plane by minimizing the weighted distance between the reconstructed plane depth map and the complete depth map. Here, the plane visibility confidence plays a key role as an important weighting factor. Specifically, when the visibility confidence of a certain area is high, it is more inclined to make the reconstructed plane depth map consistent with the complete depth map of the area to ensure the accuracy of the estimated plane depth parameters. In this way, the C2P conversion method not only considers the geometric characteristics of the plane, but also fully utilizes the visibility information of the plane, thereby achieving robust estimation of the depth of the plane in indoor scenes.

[0056] C2P transformation in perspective images:

[0057] Step 1: Use a parametric method to represent the depth of a plane in a perspective image.

[0058] The following is a brief introduction to parameterization. According to the pinhole camera model, the projection of a 3D point (x, y, z) from the camera coordinate system to a pixel (u, v) in the perspective image is as follows:

[0059]

[0060] where f x , f y is the focal length, (u0, v0) is the principal point. If the equation is If any plane is projected onto the image, its pixel coordinates and depth values ​​will satisfy This equation can be restated as:

[0061]

[0062] Here, Defined as the depth parameter of a plane in a perspective image. The definition of depth in a perspective image (distance along the z-axis) can be parameterized in this way.

[0063] Step 2: Based on the plane depth parameterization, C2P transformation is introduced. In view of this, the C2P transformation of perspective images is similar to that of panoramic images. The goal of C2P transformation is to derive the representation of P k The plane depth parameter of plane k is obtained, thereby facilitating plane depth reconstruction. Taking plane k as an example, let Z represent the complete depth map of the perspective image, represents the plane visibility confidence map, Q k Denotes the target plane depth parameter of plane k. It is constructed using the homogeneous pixel coordinates obtained from Z and their corresponding inverse depth values. and The formula is as follows:

[0064]

[0065] To ensure consistency, d i To represent the inverse depth value of the i-th pixel in the perspective image, that is Second, use the elements on the diagonal Construct a diagonal matrix Using this method, the target plane depth parameter Q is calculated k as follows:

[0066]

[0067] For planar depth map reconstruction, construct and in

[0068] Given The reconstructed depth map of plane k is calculated as follows:

[0069]

[0070] Notably, the C2P transformation is differentiable for perspective images, allowing the gradients of the loss function applied to the estimated plane depth parameters and the reconstructed plane depth map to be back-propagated to the plane visibility confidence.

[0071] Perspective Image Frame:

[0072] Step 3: Design of perspective image; Figure 2As shown, the architecture of the designed perspective image. Perspective images with limited viewing angles usually only capture a portion of the room. The organization of dominant planes in perspective images lacks the regularity in panoramic images. In order to classify these planes, M×M anchor points are defined that are uniformly distributed on the perspective image. For each dominant plane (including the ceiling and floor), the centroid coordinates of the corresponding segmented region are calculated. The label assigned to the plane corresponds to the index of the anchor point closest to the centroid. Specifically, the plane whose region centroid is closest to the anchor point in the i-th row and j-th column obtains a plane k, where k = (i-1)M+j. Through this labeling strategy, a total of M are defined. 2 Different plane categories.

[0073] To constrain the estimation of plane depth parameters, we use a weighted L1 loss function between the estimated plane depth parameters and the ground truth. Considering that the plane depth parameters of larger plane areas have greater importance on the layout estimation results, we adjust the weight of the loss function according to the number of pixels in the layout segmentation mask of each plane:

[0074]

[0075] Therefore, the output branch responsible for predicting the plane category produces a probability map that is aligned with the distribution space of the anchors If there is a plane k in the image, then The i-th row and j-th column (expressed as ) should be 1; otherwise, it should be 0. The learning of is supervised by the binary map generated by the ground truth layout segmentation mask, denoted as Taking into account There is a significant imbalance between positive and negative samples, using and The weighted cross entropy loss between is used as the loss function:

[0076]

[0077] where t is the weighting factor which is experimentally set to t=25.

[0078] For perspective images, due to the significant increase in plane categories, the prediction of corresponding plane semantic labels also requires more output channels. Set M = 16, indicating a total of 256 plane categories. Too many output channels may lead to overfitting, especially when dealing with limited training data. To address this problem, a decoupling method is used to reduce the output channels while predicting the plane semantic labels. In this way, the number of output channels is reduced from M to 16. 2 is reduced to 2M, where M channels are dedicated to predicting the x-axis component α∈R h×w×M , the remaining M channels are dedicated to predicting the y-axis component β∈Rh×w×M . The j-th channel mapping of α is represented as α (j) (j=1, 2, ···, M), which is a probability map indicating whether the plane class of each pixel is associated with an anchor in the jth column. Similarly, β (i) (i=1, 2, ···, M) indicates whether the plane category of each pixel is associated with one of the anchors in the i-th row. Therefore, α j and β i The joint probability of represents the semantic label of the plane associated with the anchor in the i-th row and j-th column, and the plane category is (i-1)M+j. Definition To represent M 2 All semantic labels of the plane category. Since α and β are independent, Each channel map in can be obtained by element-wise multiplication of the channel maps in α and β:

[0079]

[0080] in Indicates the right-to-speak operation. Based on the above analysis, the corresponding output branch is used to predict the decoupled plane semantic labels of the 2M output channels. The training objective of the decoupled planar semantic labels represented by is composed of the sum of the binary cross entropy loss and the dice loss specified in the formula.

[0081] We expect the predicted full depth map to vary more smoothly within each planar region while showing clear and sharp edges at the intersection of the two main planes. To this end, we adopt the multi-scale gradient matching loss function defined as follows:

[0082]

[0083] We use 4 scales and use ρ∈{1, 2, 3, 4} to represent the scale. Gρ refers to the depth difference map of scale ρ, that is, G 1 =RR * , G ρ The resolution is G ρ-1 Half of n ρ is the number of pixels of scale ρ.

[0084] While learning the full depth map, the network is trained to predict the inverse depth representation. The loss function denoted by combines the scale and translation invariant pruning loss and the multi-scale gradient matching loss in Equation (16). The loss function of the coarse layout depth map (denoted as Z) is generated by the weighted average of the planar depth maps, and its components are Same, recorded as In addition, the loss function of the estimated plane depth parameter is consistent with formula (22), which is expressed as

[0085] Finally, the overall loss function for perspective layout learning is defined as:

[0086]

[0087] Among them, u1, u2, u3, u4 and u5 are weighting coefficients.

[0088] Step 4: Sequential planar depth map integration;

[0089] After the training phase is completed, a method applicable to both panoramic and perspective images is proposed to reconstruct the 3D layout through sequential planar depth map integration.

[0090] First, the C2P conversion technology is used to generate the plane depth maps of the ceiling, floor and each wall.

[0091] The walls are then sorted from left to right, and their respective plan depth maps are integrated in a sequential, step-by-step manner. This sequential integration ensures smooth transitions and precise connections between walls, laying the foundation for an accurate 3D layout. The plan depth maps of the ceiling and floor are also taken into account during the integration process, ensuring they blend seamlessly with the wall depth maps.

[0092] Ultimately, through a meticulous integration process, a complete and accurate 3D layout depth map is obtained. Notably, this method does not rely on the assumption that walls are perpendicular to each other, making it effective in complex indoor environments that deviate from the traditional Manhattan world assumption. This feature gives the method greater flexibility and applicability, allowing it to handle a variety of complex indoor scenes. Finally, minor modifications to the perspective image are explained. This versatility further enhances the method's value in practical applications.

[0093] Eliminate duplicate predictions of the wall category based on the similarity of the corresponding semantic labels, such as Figure 3 As shown, (a) adjacent wall category predictions correspond to different walls. (b) adjacent wall category predictions are repeated, where only the category with the higher activation value is retained.

[0094] The sequential planar depth map integration specifically includes:

[0095] Step 4.1. Determine the wall category in the image. Use C2P conversion technology to generate plane depth maps of the ceiling, floor, and each wall. Use the plane category prediction C to define the threshold δ c , and select the predicted value greater than δ cHowever, this selection process may introduce duplication, as nearby predictions may potentially represent the same plane within the image. Figure 3 As shown in Figure 2, the predicted semantic labels of the adjacent plane categories are used to filter these duplicates. Whenever the wall category is selected, its two adjacent categories are checked. If any of these adjacent categories is also selected, the DICE loss is calculated to evaluate the similarity between the predicted semantic labels of these adjacent planes. The DICE loss exceeds a predefined threshold δ. d Indicates that adjacent predictions correspond to different walls, prompting both to be retained. On the contrary, when the dice loss is lower than δ d When , it is suggested to keep only the wall category with higher prediction value. c =0.3,δ d =0.5.

[0096] Step 4.2 further requires distinguishing the selected planes into walls and non-walls. This is achieved by classifying the estimated plane depth parameters. Generally, the vertical component of the surface normal is not important for walls, but is dominant for ceilings and floors. Consider a plane with estimated depth parameters Here, the second parameter σ k is proportional to the perpendicular component of the plane normal. If the ratio Below a predefined threshold δ q , we classify plane k as a wall. Otherwise, we classify plane k as a non-wall. We experimentally classify δ q Set to 0.75. After identifying all walls, we sort them from left to right based on the horizontal coordinates of the centroids associated with the plane semantic labels. A plane depth map can be viewed as a 3D surface in a coordinate system defined by pixel coordinates and a depth axis. For clarity, the integral of two plane depth maps is analyzed as the intersection of two surfaces in this 3D coordinate system. This process begins with the depth map of the leftmost wall, gradually integrating the depth maps of subsequent walls. When integrating a new wall, an intersection line is formed between the new plane depth map and the existing plane depth map. In a real room layout, the two walls should lie on either side of this intersection line. The position of each wall relative to the intersection point is determined by referencing the centroid position derived from the plane semantic label. Regions of the plane depth map that extend beyond the intersection line are trimmed, resulting in an updated layout depth map. This process is repeated until the plane depth maps of all walls have been integrated.

[0097] Step 4.3: Through a detailed integration process, a complete and accurate 3D layout depth map is obtained. Finally, the depth maps of the ceiling and floor are integrated into the layout depth map using a pixel-wise minimum operation. From the resulting layout depth map, a layout segmentation mask can be generated based on the plane class associated with each depth value.

[0098] To further enhance the robustness of the prediction, during the test phase, the input image is horizontally flipped and an alternative layout estimate (horizontally flipped) is obtained using the same method. For both the original and flipped images, the pixel-wise matching rate between each layout segmentation mask, obtained through sequential integration, and S is evaluated, and the one with the highest matching rate is selected as the final layout estimate. This approach ensures the reliability and accuracy of the prediction.

[0099] Example

[0100] Implementation of Perspective Image. The Perspective Image Network is built on a relatively simple architecture designed for perspective depth estimation. It uses SENET-154 as the encoder. The output branch for full depth map prediction consists of a decoder, a multi-scale feature fusion module, and a refinement module. The input image resolution is set to 256×256, and the output depth map resolution is 128×128. The output branch is also extended with a sigmoid activation to predict a global plane visibility confidence map and plane semantic labels. The output branch for plane category prediction begins with a 1×1 convolutional layer, also reducing the channel dimension to 256. Following this, a bilinear interpolation layer resizes the spatial resolution of the feature map to M×M. Subsequently, seven consecutive 3×3 convolutional layers are applied, with batch normalization and the ReLU activation function combined between each pair of consecutive convolutional layers. The ReLU activation function is an activation function used in deep learning that aims to address some limitations of the traditional ReLU activation function and is used in some deep learning models, particularly when processing complex datasets or requiring high model expressiveness. A sigmoid activation is applied to the output layer. The size of the flat class prediction is M×M, where M is set to 16 by default.

[0101] The ADAMP optimizer was used with a batch size of 20, an initial learning rate of 0.0001, and a 0.2% reduction every 70 epochs. Training images were augmented with random horizontal flips, random rotations within ±5 degrees, and color jittering. The weighting coefficients were experimentally set to: u1 = 2, u2 = 10, u3 = 1, u4 = 20, and u5 = 2.

[0102] Evaluation Metrics:

[0103] Evaluation metrics for perspective image-based layout estimation include i) pixel error (PE); ii) corner error (CE); iii) 3D corner error (3DCE), which represents the average distance between corresponding layout corners in the camera coordinate system; iv) root mean square error (Rmse); v) absolute relative error (Rel); vi) mean log10 error (Log10); and vii) threshold accuracy (δ1, δ2, δ3), which are δ1, δ2, 1.252, and δ3, 1.253, respectively. Metrics iv-vii are used to evaluate the predicted layout depth map against the ground truth.

[0104] Results of pivoting the dataset

[0105] We also conducted experiments on four perspective datasets: Matterport3D-Layout, InteriorNet-Layout, LSUN, and Hedau. The dataset descriptions and experimental results are shown below.

[0106] The Matterport3D-Layout dataset was also constructed by carefully selecting and annotating the Matterport3D dataset. The Matterport3DLayout dataset relies heavily on perspective images. It includes a training set of 4,939 images, a validation set of 456 images, and a test set of 1,965 images, encompassing both cuboid and non-cuboid room types.

[0107] Table I: Quantitative results of overall layout estimation based on perspective images on the Matterport3D-Layout dataset:

[0108]

[0109] Specifically, PlaneAE, GeoLayout, Zhang et al., and Yao et al. all rely on direct learning of plane parameters. Experimental results demonstrate that our approach significantly outperforms all existing methods. Compared to GeoLayout, our approach achieves 1.74%, 1.34%, and 2.46% error reduction in PE, CE, and 3DCE, respectively. Metrics related to the layout depth map also show considerable performance improvements, confirming the effectiveness of C2P transformation compared to directly learning plane parameters. Furthermore, all RGB-D perspective images of buildings corresponding to the training set were collected from the Matterport3D dataset. After filtering out some unqualified images, a total of 125,419 RGB-D images were collected. The performance of models pre-trained on this external RGBD dataset is significantly improved. Given the abundance of current indoor RGB-D perspective image datasets, this approach has significant potential for further improvement.

[0110] Visualization results of perspective layout estimation on Matterport3D-Layout and InteriorNet-Layout datasets, such as Figure 4 As shown, Figure 4 (a) shows the visualization results of our method on the Matterport 3D-Layout dataset, with GeoLayout, Planae, and Yao et al. for comparison. From the learned plane visibility confidence map, we can observe that the learned confidence scores for occluded object regions are low, and the outlines of object boundaries are clearly visible. Compared with other compared methods, our method achieves the highest accuracy and can detect planes with smaller areas, such as the tilted plane on the right side of the third column.

[0111] InteriorNet-Layout. This dataset was created by selecting and annotating images from the photorealistic synthetic dataset InteriorNet. The training, validation, and test sets consist of 6,534, 833, and 852 images, respectively. This dataset also provides comprehensive 2D and 3D layout labels, covering both cuboid and non-cuboid room types. Notably, a notable feature of the InteriorNet-Layout dataset is the abundance of multi-wall samples, with 21% of samples containing more than three walls. This abundance of multi-wall samples highlights the rich diversity of this dataset compared to other datasets.

[0112] Table II: Quantitative results of overall layout estimation based on perspective images on the InteriorNet-Layout dataset:

[0113]

[0114] This method achieves the best performance across all metrics, reducing pixel error (PE) by 3.28%, corner error (CE) by 1.48%, and 3D corner error (3DCE) by 1.69% compared to state-of-the-art methods. These results demonstrate that the advantages of this method are further amplified on datasets containing more complex multi-wall samples, demonstrating the robustness of this method in handling complex multi-wall scenes.

[0115] The visualization results of the InteriorNet-Layout dataset are as follows Figure 4(b) shows this method. Compared with other comparison methods, this method provides the most accurate layout estimation results, especially in multi-wall scenarios. It is worth noting that compared with the Matterport3D-Layout dataset, the plane visibility confidence map learned from this dataset is sparse and mainly concentrated on the boundaries of the plane. The possible reason is that InteriorNet-Layout belongs to a synthetic dataset with relatively low ground truth depth noise. This favorable condition makes the learned plane depth closer to the plane depth equation. Therefore, C2P conversion requires fewer pixels to accurately match the plane depth parameters.

[0116] LSUNandHedau. The LSUN dataset consists of 4000 training images, 394 validation images, and 1000 test images. In this dataset, all samples belong to the cuboid room type and only provide two-dimensional layout labels. Due to the lack of depth information, this dataset is not suitable for direct training of the model. Instead, training is performed on the Matterport3D-Layout dataset. Subsequently, the RGB-D images of Matterport3D-Layout are combined to fine-tune the comprehensive depth map prediction branch, while the other components of the model are refined using data from the LSUN dataset. The loss function is modified to: here Optimized using the Matterport3D-Layout dataset, and and Optimization was performed using the LSUN dataset. is the cross entropy loss between the defined scaled inverse depth soft maximum and the ground truth layout segmentation.

[0117] This term encourages that the depth value of the plane reconstructed in the corresponding plane region is the minimum value among all plane depth maps. A slight modification is made, where the ground truth layout depth map is replaced by a depth map consisting of the minimum depth value of each pixel across all predicted planar depth maps. The weight coefficients are set to η1 = 1, η2 = 10, η3 = 1, η4 = 1, and η5 = 10. Since the ground truth results on the LSUN test set are not publicly available, we adhere to the same evaluation practices as existing methods and test on the validation set.

[0118] The Hedau dataset consists of 209 training images and 105 test images, all of which are cuboid. This dataset only provides ground truth layout segmentation masks. Due to its limited size, only models trained on LSUN are tested on this dataset.

[0119] Table III: Quantitative results of cuboid layout estimation based on perspective images on LSUN and Hedau datasets:

[0120]

[0121] The performance is shown in Table III. Our method achieves the lowest pixel error and corner error on the LSUN dataset and the second-lowest pixel error on the Hedau dataset. The method's performance on the 2D layout dataset is less impressive than on the 3D dataset. This discrepancy can be attributed not only to the lack of ground-truth depth data, but also to the inconsistency between some manually annotated 2D layouts and 3D geometric constraints in the LSUN and Hedau datasets. This inconsistency becomes particularly pronounced in cases involving severe occlusions, where the annotator must rely on educated guesses to annotate the occluded parts of the layout.

[0122] Ablation

[0123] Qualitative comparison results of layout estimation and supervised plane visibility confidence, such as Figure 5 As shown, the comparison is evaluated on the MatterportLayout (top) and Matterport3DLayout (bottom) datasets. The red line is the true layout and the green line is the prediction. The main differences are highlighted with dotted lines.

[0124] Table IV: Quantitative comparison of perspective layout estimation with planar visibility confidence supervision on the Matterport3D-Layout dataset:

[0125]

[0126] In this method, plane visibility confidence is learned autonomously without direct supervision. To test the effectiveness of this learning strategy, pseudo-ground truth values ​​of plane visibility confidence are generated based on the pixel-wise similarity between the true global depth map and the layout depth map. As a comparison method, an L1 loss between the predicted global plane visibility confidence and the pseudo-ground truth is incorporated into the training process. Qualitative comparisons of supervised and unsupervised plane visibility confidence on the MatterportLayout and Matterport3D-Layout datasets are shown in Tables IV and V, respectively. The results show that methods without direct supervision show better performance. Figure 5 A qualitative comparison is presented. It can be seen that although the plane visibility confidence map learned through direct supervision covers most of the visible plane area, the final layout estimate is not as accurate as the map obtained through independent learning. This shows that forcing the network to fit all visible plane areas is unnecessary. Instead, allowing the network to independently identify and focus on the most confident visible plane areas leads to more favorable results.

[0127] Qualitative comparison results of the learned complete depth map and the layout depth map, such as Figure 6 As shown. Evaluation comparison on the MatterportLayout (top) and Matterport3D-Layout datasets, the red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines.

[0128] Table V: Quantitative comparison of perspective layout estimation using learned layout depth maps and full depth maps on the Matterport3D-Layout dataset:

[0129]

[0130] During C2P-Net training, depth information is obtained for occluded portions of the full depth map. This helps the network gain a comprehensive understanding of the overall spatial structure of indoor scenes. To verify the impact of learning depth information for occluded portions, a comparative experiment was conducted in which the branch originally used to predict the full depth map was used to predict the layout depth map. Tables IV and V provide quantitative comparison results on the MatterportLayout and Matterport3D-Layout datasets, respectively. The results show that learning the full depth map helps obtain better layout estimates, demonstrating that learning depth information for occluded portions is beneficial for enhancing the network's understanding of room layout. Figure 6 Qualitative comparison results are presented. The results show that, compared to learning a layout depth map, a network trained on a full depth map is able to obtain plane visibility confidence maps that are able to distinguish planes from obstacles to a certain extent. This difference is even more pronounced on the perspective dataset Matterport3D-Layout. This suggests that learning a full depth map enables the network to automatically distinguish between obstacles and planes, thereby gaining a deeper understanding of the spatial structure of the scene. As a result, the final layout estimate is more accurate.

[0131] Table VI: Quantitative comparison of perspective layout estimation for different M values ​​on the Matterport3DLayout dataset:

[0132]

[0133] In C2P-Net, the number of plane categories (i.e., M) used for perspective images is a key hyperparameter. The increase in the number of categories improves the network's ability to distinguish dense narrow planes, thereby pushing up the upper limit of the accuracy that the method can achieve. However, this expansion in categories also increases the challenge of accurate classification and expands the channels for learning plane semantic labels. As a result, some channels become prone to over-adaptation due to insufficient training. In Table VI, the perspective layout estimation results for different M values ​​are evaluated on the Matterport3D-Layout dataset. The model with M=16 has the best overall performance.

[0134] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A perspective view C2P conversion and layout generation method for room layout estimation, comprising: Step 1. Use a parametric method to represent the depth of a plane in a perspective image. According to the pinhole camera model, the projection of a 3D point (x, y, z) from the camera coordinate system to a pixel (u, v) in the perspective image is as follows: where f x , f y is the focal length, (u0, v0) is the principal point, if the equation is If any plane is projected onto the image, its pixel coordinates and depth values ​​will satisfy This equation can be reformulated as: Here, It is defined as the planar depth parameter of a plane in a perspective image; is the normal vector, is the distance to the camera center; σ, are the three parameters that define the depth of the plane; Step 2: Based on the plane depth parameterization, C2P conversion is introduced; Step 3: Design of perspective image; Classify the planes and define M×M anchor points evenly distributed on the perspective image. For each dominant plane including the ceiling and floor, calculate the centroid coordinates of the corresponding segmented region. The label assigned to the plane corresponds to the index of the anchor point closest to the centroid. Step 4: Sequential planar depth map integration; The step 2 introduces C2P conversion, which includes: the goal of C2P conversion is to derive the expression P k The plane depth parameter of plane k is used to facilitate plane depth reconstruction; let plane be k, let Z represent the complete depth map of the perspective image, Represents the plane visibility confidence, Q k Denotes the target plane depth parameter of plane k, constructed using the homogeneous pixel coordinates obtained from Z and their corresponding inverse depth values. and The formula is as follows: To ensure consistency, d i To represent the inverse depth value of the i-th pixel in the perspective image, that is Second, use the elements on the diagonal Construct a diagonal matrix Calculate the target plane depth parameter Q k as follows: For planar depth map reconstruction, construct and in Given The reconstructed depth map of plane k is calculated as follows: Z represents the comprehensive depth map of the perspective image, and the homogeneous pixel coordinates obtained by Z and their corresponding inverse depth values ​​are constructed respectively. and matrix; n is the total number of pixels in the image; u n 、v n Represents the horizontal and vertical coordinates of the nth pixel; d n Represents the inverse depth value of the nth pixel; Zi represents the depth value of the i-th image; σ, are the three parameters that define the depth of the plane; A parameter representing the plane depth of the K-th plane; The C2P transformation is differentiable for perspective images, allowing the gradients of the loss function applied to the estimated plane depth parameters and the reconstructed plane depth map to be back-propagated to the plane visibility confidence.

2. The perspective image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The plane visibility confidence is learned autonomously without direct supervision, i.e., forcing the network to fit all visible plane regions is unnecessary; instead, letting the network independently identify and focus on the most credible visible plane regions leads to more favorable results.

3. The perspective image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 3 further includes: obtaining a plane k where the region centroid is closest to the anchor point in the i-th row and j-th column, wherein k=(i-1)M+j. By this labeling strategy, a total of M 2 different plane categories; the output branch responsible for predicting the plane category produces a probability map aligned with the distribution space of the anchors If there is a plane k in the image, then The i-th row and j-th column of The value in should be 1, otherwise it should be 0; The learning of is supervised by the binary map generated by the ground truth layout segmentation mask, denoted as Taking into account There is a significant imbalance between positive and negative samples, using and The weighted cross entropy loss between is used as the loss function: where t is the weighting factor set to t = 25 in the experiment; k represents the label of the plane; The output branch responsible for predicting the plane category produces a probability map that is aligned with the distribution space of the anchor; The output branch for the true plane class produces a probability map that is aligned with the distribution space of the anchors.

4. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 3, characterized in that: The step 3 further includes: using a decoupling method, the number of output channels is increased from M 2 is reduced to 2M, where M channels are dedicated to predicting the x-axis component α∈R h×w×M , the remaining M channels are dedicated to predicting the y-axis component β∈R h×w×M , the j-th channel mapping of α is represented as α (j) (j=1, 2, ···, M), which is a probability map indicating whether the plane category of each pixel is associated with an anchor in the j-th column; similarly, β (i) (i=1, 2, ···, M) indicates whether the plane category of each pixel is associated with one of the anchors in the i-th row, so α j and β i The joint probability represents the semantic label of the plane associated with the anchor in the i-th row and j-th column, and the plane category is (i-1)M+j, defined as To represent M 2 All semantic labels of the plane category, since α and β are independent, Each channel map in is obtained by element-wise multiplication of the channel maps in α and β: in Indicates the right-to-speak operation; Based on the above analysis, the corresponding output branch is used to predict the decoupled plane semantic labels of the 2M output channels. The training objective of the decoupled planar semantic label represented by is composed of the sum of the binary cross entropy loss and the dice loss specified in the formula; α (j) (j=1, 2, ···, M) is a probability map indicating whether the plane class of each pixel is associated with the anchor in the jth column; β (i) (i=1, 2, ···, M) represents the probability map of whether the plane category of each pixel is associated with the anchor in the i-th row; α j and β i represents the joint probability of the semantic labels of the plane associated with the anchor point in the i-th row and j-th column.

5. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 4 further comprises: Step 4.1: Determine the wall categories in the image and generate planar depth maps of the ceiling, floor, and each wall using C2P conversion technology. Step 4.2, distinguishing the selected plane into walls and non-walls by classifying the estimated plane depth parameters; Step 4.3: Through a detailed integration process, a complete and accurate 3D layout depth map is obtained.

6. The perspective image C2P conversion and layout generation method for room layout estimation according to claim 5, characterized in that: The step 4.1 further includes: using the plane category prediction C, defining a threshold δ c , and select the predicted value greater than δ c Whenever the wall category is selected, its two adjacent categories are checked. If any of these adjacent categories is selected, the dice loss is calculated to evaluate the similarity between the predicted semantic labels of these adjacent planes. The dice loss exceeds a predefined threshold δ d Indicates that adjacent predictions correspond to different walls, prompting both to be retained. On the contrary, when the dice loss is lower than δ d , it prompts to keep only the wall category with higher prediction value.

7. The perspective image C2P conversion and layout generation method for room layout estimation according to claim 5, characterized in that: The step 4.2 further comprises: having an estimated plane depth parameter The selected plane k, the second parameter σ k is proportional to the perpendicular component of the plane normal if the ratio Below a predefined threshold δ q , classify plane k as a wall, otherwise, classify plane k as a non-wall; after distinguishing all the walls, sort them from left to right based on the horizontal coordinates of the centroid associated with the plane semantic label, and the plane depth map is regarded as a 3D surface in the coordinate system defined by the pixel coordinates and the depth axis; Represents the plane depth parameter of the defined k-th plane; The formula is: σ k is the second parameter of the plane parameters of the k-th plane.

8. The perspective image C2P conversion and layout generation method for room layout estimation according to claim 5, characterized in that: The step 4.2 further includes: analyzing the integral of the two planar depth maps as the intersection of the two surfaces in the 3D coordinate system, starting the process with the depth map of the leftmost wall, and gradually integrating the depth maps of subsequent walls. When a new wall is integrated, an intersection line is formed between the new planar depth map and the existing planar depth map; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label, trimming the area of ​​the planar depth map that extends beyond the intersection line, thereby generating an updated layout depth map. This process will be repeatedly applied until the planar depth maps of all walls are integrated.

9. The perspective image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The perspective image network is built on the architecture designed for perspective depth estimation, which uses SENET-154 as the encoder. The output branch for full depth map prediction includes a decoder, a multi-scale feature fusion module, and a refinement module.

Images