Room view C2P conversion and layout generation method for room layout estimation

Abstract

The invention discloses a perspective C2P conversion and layout generation method for room layout estimation. Comprising the steps of obtaining a target indoor scene image; inputting a target indoor scene image into the deep learning model, and outputting a predicted complete depth map, a global plane visibility confidence map, a wall category and a plane semantic tag; the method comprises the following steps of: classifying a plane, defining anchor points uniformly distributed on a perspective image, calculating centroid coordinates of a corresponding segmentation region of a dominant plane for the dominant plane, and obtaining a rough layout segmentation mask by a label assigned to the plane corresponding to an index of the anchor point closest to the centroid; a plane depth parameter is generated from the complete depth map and the plane visibility confidence. A planar depth map is generated using a C2P conversion technique, and the planar layout of a room is accurately estimated through sequential planar depth map integration. According to the method, C2P transformation can be flexibly applied to perspective images, and powerful technical support is provided for plane depth reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a perspective view C2P conversion and layout generation method for room layout estimation. Background Art

[0002] Room layout estimation is a challenging problem in computer vision, which aims to infer the overall spatial layout of indoor environments from perspective images. Applications include scene reconstruction, indoor localization, and augmented reality.

[0003] Room layout is determined by major indoor planes, especially the floor, ceiling, and walls. Recovering 3D room layout essentially involves reconstructing these major indoor planes, which is a fundamental challenge in layout estimation. Some recent studies reconstruct these major planes in perspective images by learning plane parameters. However, a significant limitation of these methods is the lack of a mechanism to guide the network to focus on the visible regions of major indoor planes, which makes the prediction susceptible to occlusions. In addition, the determination of plane parameters is affected not only by the visual appearance but also by the positional context of the planes in the image. This dual dependency increases the complexity and imposes an additional burden on network fitting. Summary of the invention

[0004] The object of the present invention is to provide a perspective view C2P conversion and layout generation method for room layout estimation, so as to solve the technical problems existing in the existing prior art.

[0005] To achieve the above object, the perspective view C2P conversion and layout generation method for room layout estimation of the present invention includes:

[0006] Step 1: Use a parametric method to represent the plane depth in the perspective image; according to the pinhole camera model, the projection of the 3D point (x, y, z) from the camera coordinate system to the pixel (u, v) in the perspective image is as follows:

[0007]

[0008] where f x , f y is the focal length, (u0, v0) is the principal point, if we change the equation to If any plane is projected onto the image, its pixel coordinates and depth values ​​will satisfy This equation can be restated as:

[0009]

[0010] Here, Defined as the planar depth parameter of a plane in a perspective image;

[0011] Step 2: Based on the plane depth parameterization, C2P conversion is introduced;

[0012] Step 3, design of perspective image; classify the planes, define M×M anchor points evenly distributed on the perspective image, and for each dominant plane including the ceiling and the floor, calculate the centroid coordinates of the corresponding segmented area, and the label assigned to the plane corresponds to the index of the anchor point closest to the centroid;

[0013] Step 4: Sequential planar depth map integration.

[0014] Further, the step 2 introduces C2P conversion including: the goal of C2P conversion is to derive k The plane depth parameter of plane k is obtained, thereby facilitating plane depth reconstruction; let the plane be k, let Z represent the complete depth map of the perspective image, represents the plane visibility confidence, Qk represents the target plane depth parameter of plane k, and is constructed using the homogeneous pixel coordinates obtained from Z and its corresponding inverse depth value and The formula is as follows:

[0015]

[0016] To ensure consistency, di is also used to represent the inverse depth value of the i-th pixel in the perspective image, that is, Second, use the elements on the diagonal Construct a diagonal matrix Calculate the target plane depth parameter Q k as follows:

[0017]

[0018] For planar depth map reconstruction, construct and in Given The reconstructed depth map of plane k is calculated as follows:

[0019]

[0020] The goal of C2P transformation is to derive a k The C2P transformation is differentiable for perspective images, allowing the gradient of the loss function applied to the estimated plane depth parameters and the reconstructed plane depth map to be back-propagated to the plane visibility confidence.

[0021] Furthermore, the plane visibility confidence is learned autonomously without direct supervision, i.e., it is unnecessary to force the network to fit all visible plane regions. Instead, letting the network independently identify and focus on the most credible visible plane regions will lead to more favorable results.

[0022] Furthermore, the step 3 further comprises: obtaining a plane k whose region centroid is closest to the anchor point in the i-th row and j-th column, where k = (i-1)M+j. Through this labeling strategy, a total of M 2 different plane categories; the output branch responsible for predicting the plane category produces a probability map that is aligned with the distribution space of the anchors If there is a plane k in the image, then The i-th row and j-th column of The value in should be 1, otherwise it should be 0; The learning of is supervised by a binary map generated from the ground truth layout segmentation mask, denoted as Considering There is a significant imbalance between positive and negative samples, using and The weighted cross entropy loss between is used as the loss function:

[0023]

[0024] where t is the weighting factor which is experimentally set to t=25.

[0025] Furthermore, the step 3 further comprises: using a decoupling method, the number of output channels is increased from M to 2 is reduced to 2M, where M channels are dedicated to predicting the x-axis component α∈R h×w×M , the remaining M channels are dedicated to predicting the y-axis component β∈R h×w×M , the j-th channel mapping of α is represented as α (j) (j=|, 2, ..., M), which is a probability map indicating whether the plane class of each pixel is associated with an anchor in the jth column; similarly, β (i) (i = |, 2, ..., M) indicates whether the plane category of each pixel is associated with one of the anchors in the i-th row, so α j and β i The joint probability represents the semantic label of the plane associated with the anchor in the i-th row and j-th column. The plane category is (i-1)M+j, defined as To represent all semantic labels of M2 plane categories, since α and β are independent, Each channel map in can be obtained by element-wise multiplication of the channel maps in α and β:

[0026]

[0027] in Indicates the right-to-speak operation; Based on the above analysis, the corresponding output branch is used to predict the decoupled plane semantic labels of the 2M output channels. The training objective of the decoupled planar semantic labels represented by is composed of the sum of the binary cross entropy loss and the dice loss specified in the formula.

[0028] Furthermore, the step 4 further comprises:

[0029] Step 4.1, determine the wall category in the image, and use the C2P conversion technology to generate the plane depth map of the ceiling, floor and each wall;

[0030] Step 4.2, distinguishing the selected plane into walls and non-walls by classifying the estimated plane depth parameters;

[0031] Step 4.3: Through a fine integration process, a complete and accurate 3D layout depth map is obtained.

[0032] Furthermore, the step 4.1 further includes: using the plane category prediction C, defining a threshold δ c , and select the predicted value greater than δ c plane category; whenever the wall category is selected, its two neighboring categories are checked. If any of these neighboring categories is selected, the dice loss is calculated to evaluate the similarity between the predicted semantic labels of these neighboring planes. The dice loss exceeds a predefined threshold δ d Indicates that adjacent predictions correspond to different walls, encouraging both to be retained. On the contrary, when the dice loss is lower than δ d , it is suggested to keep only the wall categories with higher prediction values.

[0033] Further, the step 4.2 further comprises: having an estimated plane depth parameter The selected plane k, the second parameter σ k is proportional to the perpendicular component of the plane normal if the ratio Below a predefined threshold δ q , classify plane k as a wall, otherwise, classify plane k as a non-wall; after distinguishing all the walls, they are sorted from left to right based on the horizontal coordinates of the centroids associated with the plane semantic labels. The plane depth map is regarded as a 3D surface in a coordinate system defined by pixel coordinates and depth axis.

[0034] Furthermore, the step 4.2 further includes: analyzing the integral of the two plane depth maps as the intersection of the two surfaces in the 3D coordinate system, starting the process with the depth map of the leftmost wall, gradually integrating the depth maps of subsequent walls, and when integrating new walls, an intersection line is formed between the new plane depth map and the existing plane depth map; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label, trimming the area of ​​the plane depth map extending beyond the intersection line, thereby generating an updated layout depth map, and this process will be repeatedly applied until the plane depth maps of all walls are integrated.

[0035] Furthermore, the perspective image network is built on top of the architecture designed for perspective depth estimation, which adopts SENET-154 as the encoder, and the output branch for full depth map prediction includes a decoder, a multi-scale feature fusion module, and a refinement module.

[0036] The method of the present invention has the following advantages:

[0037] 1) The present invention introduces the C2P transform, which is a differentiable operation method specifically used to achieve robust planar depth reconstruction. This method is based on the derived planar depth parameter representation, which enables the C2P transform to be flexibly applied to perspective images and provides strong technical support for planar depth reconstruction.

[0038] 2) This paper proposes an efficient and versatile 3D layout estimation learning framework that is well suited for perspective images. Based on the prediction results of the network, an innovative sequential planar depth map integration method is further proposed to generate accurate 3D layouts. This method not only improves the accuracy and reliability of 3D layout estimation, but also demonstrates the strong potential of the framework in processing different types of images.

[0039] 3) The proposed method achieves state-of-the-art performance in all aspects of layout estimation benchmarks of perspective images.

[0040] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the C2P conversion graph;

[0042] Figure 2 The C2P network diagram for perspective images;

[0043] Figure 3Schematic diagram of eliminating repeated predictions of wall categories based on the similarity of corresponding semantic labels. (a) Adjacent wall category predictions correspond to different walls. (b) Repeated parts of adjacent wall category predictions, where only categories with higher activation values ​​are retained.

[0044] Figure 4 A diagram showing the visualization results of perspective layout estimation on the Matterport3D-Layout and InteriorNet-Layout datasets;

[0045] Figure 5 Schematic diagram of qualitative comparison results of layout estimation and supervision plane visibility confidence;

[0046] Figure 6 Schematic diagram of the qualitative comparison results of learning the complete depth map and the layout depth map, evaluated on the Matterport3D-Layout dataset. The red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines. DETAILED DESCRIPTION

[0047] The technical scheme of the present invention will be clearly and completely described below in conjunction with specific embodiments, but those skilled in the art should understand that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] This paper provides an innovative room layout estimation method, the core of which is plane depth reconstruction. First, a parametric expression of plane depth in perspective images is derived. Then, based on this foundation, a complete depth map to plane depth (C2P) conversion method is proposed, which can be effectively applied to perspective images. Figure 1As shown in the figure, through C2P conversion, the plane depth of the wall outlined in the image is reconstructed using the complete depth map of the wall and the plane visibility confidence. By using the parametric expression of the plane depth, the C2P conversion method transforms the plane depth reconstruction problem into the prediction of the comprehensive depth map and the plane visibility confidence. Specifically, the complete depth map is responsible for capturing the depth information of the visible plane area and the occluded objects, while the visibility confidence of the plane reflects the probability that the plane is observed at each pixel position, that is, it is not occluded by indoor objects or other significant planes. After obtaining the predicted complete depth map and plane visibility confidence, the weighted least squares method is further used for solving. This solution process is based on a premise: the reconstructed plane depth should be closely aligned with the complete depth map in the area with high plane visibility confidence. In this way, the plan layout of the room can be accurately estimated. The detailed implementation steps and principles of C2P conversion of perspective images will be described in detail later.

[0049] Based on the C2P transformation method, this paper further proposes an efficient learning framework for accurate 3D layout estimation. The framework tailors two distinctive architectures for perspective images, aiming to simultaneously learn comprehensive depth maps and plane visibility confidences. Due to the differentiability of the C2P transformation, necessary constraints are imposed on the predicted plane depth parameters and the reconstructed plane depth map, which enables the network to autonomously learn and optimize the plane visibility confidence as needed. In the testing phase, an innovative sequential plane depth map integration method is introduced, which can be effectively applied to perspective images to generate high-quality 3D layouts. By learning a comprehensive depth map including planes and objects, the network is able to fully understand the overall spatial structure of indoor scenes. At the same time, the joint learning of plane visibility confidences gives the network the ability to detect occlusions and focus on the visible area of ​​the dominant plane. Compared with the traditional method of directly learning plane parameters, this method shows higher robustness in dealing with occlusions. In addition, the network prediction does not depend on the specific location, which directly solves the key problem of plane parameter learning. Although extending labeled 3D layout datasets is costly, our method is able to fully leverage rich external RGB-D datasets to enhance its performance, further improving its practicality and generalization ability.

[0050] Room layout estimation based on perspective images:

[0051] Many traditional layout estimation methods adopt a classic framework, first generating a series of layout hypotheses based on detected vanishing points / lines or other image features, and then using a scoring function to evaluate these hypotheses and select the best one. Some methods also use 3D priors and 3D reasoning for layout reasoning, such as classifying surface directions based on vanishing lines and modeling the joint distribution of room layout and indoor objects. With the development of deep learning, many new methods have been introduced, one of which uses deep networks to predict two-dimensional layout representations as references and infer parameterized layout results. Many studies infer layouts by learning probabilistic maps representing the boundaries between major planes (i.e., ceilings, floors, and walls). Another class of methods involves learning semantic masks associated with dominant planes. Planar semantic masks provide richer layout information than boundaries, but learning may be affected by label ambiguity caused by the definition of walls. In addition, there are some methods that directly learn the location of layout corners through one or more probabilistic maps. However, these networks have encountered challenges in mastering basic 3D knowledge about room layouts by learning such 2D representations alone.

[0052] In recent years, 3D representation and reasoning in layout estimation have attracted increasing attention. One approach assumes that the room is a 3D template cube, thereby converting the layout estimation task into camera pose estimation. The study by Lin et al. adopts the 3D template cube assumption and uses the intermediate layout representation as a bridge to learn the camera pose, thereby facilitating the recovery of the 3D layout. Ren et al. use a reinforcement learning algorithm that gradually moves the camera inside the cube to recover the camera pose. Due to the assumption of the template cube, this type of method has limited generalization ability, especially for non-cuboid rooms. Another class of methods estimates the spatial configuration of the main indoor planes and reconstructs the 3D layout through the intersection of these planes. Zhang et al. proposed to learn the pixel-level plane depth parameters of the dominant planes and cluster these parameters to generate a plane depth map for 3D layout reconstruction. Stekovic et al. used PlaneRCNN to detect plane areas and plane parameters, and then inferred the 3D layout by solving a constrained discrete optimization problem. Yang et al. reconstructed 3D non-cuboid layouts based on the detected plane bounding boxes and the jointly learned plane parameters. In addition, some studies focused on 3D reasoning, such as the orthogonality of adjacent planes. In addition, single-view Figure 3 The task of 3D plane reconstruction is to detect planar surfaces of indoor scenes including dominant planes and indoor objects and recover their planar parameters.

[0053] Plane depth reconstruction formula:

[0054] The present invention provides a C2P (complete depth map to plane depth) conversion method for perspective images. The core of the method is to generate plane depth parameters from a complete depth map and plane visibility confidence. The plane visibility confidence here is a key indicator, which represents the probability that the plane is visible at each pixel position, that is, the probability that the position is not blocked by indoor objects or other major planes. In indoor scenes, areas with high visibility confidence are often parts where there are no obstacles and plane features can be clearly observed. Based on this understanding, it is expected that the plane depth map reconstructed using the estimated plane depth parameters can maintain a close alignment relationship with the complete depth map of these areas. To achieve this goal, an optimization strategy is adopted, that is, to determine the depth parameters of each plane by minimizing the weighted distance between the reconstructed plane depth map and the complete depth map. Here, the plane visibility confidence plays a key role as an important weighting factor. Specifically, when the visibility confidence of a certain area is high, it is more inclined to keep the reconstructed plane depth map consistent with the complete depth map of the area to ensure the accuracy of the estimated plane depth parameters. In this way, the C2P conversion method not only considers the geometric characteristics of the plane, but also makes full use of the visibility information of the plane, thereby achieving a robust estimation of the depth of the plane in indoor scenes.

[0055] C2P transformation in perspective images:

[0056] Step 1: Use a parametric method to represent the plane depth in the perspective image;

[0057] Here is a brief introduction to the parameterization. According to the pinhole camera model, the projection of a 3D point (x, y, z) from the camera coordinate system to a pixel (u, v) in the perspective image is as follows:

[0058]

[0059] where f x , f y is the focal length, (u0, v0) is the principal point. If we change the equation to If any plane is projected onto the image, its pixel coordinates and depth values ​​will satisfy This equation can be restated as:

[0060]

[0061] Here, Defined as the depth parameter of a plane in a perspective image. The definition of depth in a perspective image (distance along the z-axis) can be parameterized in this way.

[0062] Step 2: Based on the plane depth parameterization, the C2P transformation is introduced. In view of this, the C2P transformation of perspective images is similar to the C2P transformation of panoramic images. The goal of the C2P transformation is to derive the representation of P k The plane depth parameters of plane k are obtained, thereby facilitating plane depth reconstruction. Taking plane k as an example, let Z represent the complete depth map of the perspective image, represents the plane visibility confidence map, Qk represents the target plane depth parameter of plane k. The homogeneous pixel coordinates obtained from Z and their corresponding inverse depth values ​​are used to construct and The formula is as follows:

[0063]

[0064] To ensure consistency, d i To represent the inverse depth value of the i-th pixel in the perspective image, that is Second, use the elements on the diagonal Construct a diagonal matrix Using this method, the target plane depth parameter Q is calculated k as follows:

[0065]

[0066] For planar depth map reconstruction, construct and in Given The reconstructed depth map of plane k is calculated as follows:

[0067]

[0068] Notably, the C2P transformation is differentiable for perspective images, allowing the gradients of the loss function applied to the estimated plane depth parameters and the reconstructed plane depth map to be back-propagated to the plane visibility confidence.

[0069] Perspective Image Frame:

[0070] Step 3: Design of perspective image; Figure 2As shown, the architecture of the designed perspective image. Perspective images with limited viewing angles usually capture only a portion of the room. The organization of dominant planes in perspective images lacks the regularity in panoramic images. In order to classify these planes, M×M anchor points uniformly distributed on the perspective image are defined. For each dominant plane (including the ceiling and the floor), the centroid coordinates of its corresponding segmented region are calculated. The label assigned to the plane corresponds to the index of the anchor point closest to the centroid. Specifically, the plane whose region centroid is closest to the anchor point in the i-th row and j-th column obtains a plane k, where k=(i-1)M+j. Through this labeling strategy, a total of M are defined. 2 Different plane categories.

[0071] To constrain the estimation of plane depth parameters, we use a weighted L1 loss function between the estimated plane depth parameters and the ground truth. Considering that plane depth parameters of larger plane areas have greater importance on the layout estimation results, we adjust the weight of the loss function according to the number of pixels in the layout segmentation mask of each plane:

[0072]

[0073] Therefore, the output branch responsible for predicting the plane category produces a probability map that is aligned with the distribution space of the anchors If there is a plane k in the image, then The i-th row and j-th column (expressed as ) should be 1; otherwise it should be 0. The learning of is supervised by a binary map generated from the ground truth layout segmentation mask, denoted as Considering There is a significant imbalance between positive and negative samples, using and The weighted cross entropy loss between is used as the loss function:

[0074]

[0075] where t is the weighting factor which is experimentally set to t=25.

[0076] For perspective images, due to the substantial increase in plane categories, the prediction of the corresponding plane semantic labels also requires more output channels. Set M = 16, indicating a total of 256 plane categories. Too many output channels may lead to overfitting, especially when dealing with limited training data. To address this problem, a decoupling method is used to reduce the output channels while predicting the plane semantic labels. In this way, the number of output channels is reduced from M to 16. 2 is reduced to 2M, where M channels are dedicated to predicting the x-axis component α∈R h×w×M , the remaining M channels are dedicated to predicting the y-axis component β∈Rh×w×M . The j-th channel mapping of α is represented as α (j) (j=1, 2, ..., M), which is a probability map indicating whether the plane class of each pixel is associated with an anchor in the jth column. Similarly, β (i) (i=1, 2, ..., M) indicates whether the plane class of each pixel is associated with one of the anchors in the i-th row. Therefore, α j and β i The joint probability of represents the semantic label of the plane associated with the anchor in the i-th row and j-th column, and the plane category is (i-1)M+j. Definition To represent M 2 All semantic labels of the plane category. Since α and β are independent, Each channel map in can be obtained by element-wise multiplication of the channel maps in α and β:

[0077]

[0078] in Represents the right-to-speak operation. Based on the above analysis, the corresponding output branches are used to predict the decoupled plane semantic labels of the 2M output channels. The training objective of the decoupled planar semantic labels represented by is composed of the sum of the binary cross entropy loss and the dice loss specified in the formula.

[0079] We expect the predicted full depth map to vary more smoothly within each planar region while showing clear and sharp edges at the intersection of the two main planes. To this end, we adopt a multi-scale gradient matching loss function defined as follows:

[0080]

[0081] We use 4 scales and use ρ∈{1, 2, 3, 4} to represent the scale. Gρ refers to the depth difference map of scale ρ, i.e., G 1 =RR * , G ρ The resolution is G ρ-1 Half of n ρ is the number of pixels of scale ρ.

[0082] While learning the full depth map, the network is trained to predict the inverse depth representation. The loss function denoted by combines the scale- and translation-invariant pruning loss and the multi-scale gradient matching loss in Eq. (16). The loss function of the coarse layout depth map (denoted as Z) is generated by weighted averaging of the planar depth maps, and its components are The same, denoted as In addition, the loss function of the estimated plane depth parameter is consistent with formula (22), which is expressed as

[0083] Finally, the overall loss function for perspective layout learning is defined as:

[0084]

[0085] Among them, u1, u2, u3, u4 and u5 are weighting coefficients.

[0086] Step 4: Sequential planar depth map integration;

[0087] After the training phase is completed, a method applicable to both panoramic and perspective images is proposed to reconstruct the 3D layout through sequential planar depth map integration.

[0088] First, the C2P conversion technology is used to generate the plane depth map of the ceiling, floor and each wall.

[0089] Subsequently, the walls are sorted from left to right and their respective plan depth maps are integrated in a sequential, step-by-step manner. This sequential integration ensures smooth transitions and precise connections between walls, laying the foundation for building an accurate 3D layout. During the integration process, the plan depth maps of the ceiling and floor are also taken into account to ensure that they are perfectly integrated with the wall depth maps.

[0090] Finally, through a delicate integration process, a complete and accurate 3D layout depth map is obtained. It is worth mentioning that this method does not rely on the assumption that walls are perpendicular to each other, so it can effectively handle complex indoor environments that deviate from the traditional Manhattan world assumption. This feature makes the method more flexible and applicable to various complex indoor scenes. Finally, the slight modification of the perspective image will be explained. This versatility further enhances the value of this method in practical applications.

[0091] The repeated predictions of the wall category are eliminated based on the similarity of the corresponding semantic labels, such as Figure 3 As shown, (a) adjacent wall category predictions correspond to different walls. (b) adjacent wall category predictions are repeated, where only the category with higher activation value is retained.

[0092] The sequential planar depth map integration specifically includes:

[0093] Step 4.1. Determine the wall category in the image. Use C2P conversion technology to generate plane depth maps of the ceiling, floor, and each wall. Use the plane category prediction C to define the threshold δ c , and select the predicted value greater than δ c However, this selection process may introduce duplication since nearby predictions may potentially represent the same plane within the image. Figure 3 As shown in Figure 2, the predicted semantic labels of the adjacent plane categories are used to filter these duplicates. Whenever the wall category is selected, its two adjacent categories are checked. If any of these adjacent categories are also selected, the DICE loss is calculated to evaluate the similarity between the predicted semantic labels of these adjacent planes. The DICE loss exceeds a predefined threshold δ d indicates that adjacent predictions correspond to different walls, encouraging both to be retained. On the contrary, when the dice loss is lower than δ d When , it is suggested to keep only the wall category with higher prediction value. c =0.3,δ d =0.5.

[0094] Step 4.2. It is further necessary to distinguish the selected plane into walls and non-walls. This is achieved by classifying the estimated plane depth parameter. Usually, the vertical component of the surface normal is not important for walls, but is dominant for ceilings and floors. Consider a plane with an estimated depth parameter Here, the second parameter σ k is proportional to the perpendicular component of the plane normal. If the ratio Below a predefined threshold δ q , we classify plane k as a wall. Otherwise, we classify plane k as a non-wall. We experimentally q is set to 0.75. After distinguishing all the walls, we sort them from left to right based on the horizontal coordinates of the centroids associated with the plane semantic labels. A plane depth map can be viewed as a 3D surface in a coordinate system defined by pixel coordinates and a depth axis. For clarity, the integral of two plane depth maps is analyzed as the intersection of two surfaces in this 3D coordinate system. The process begins with the depth map of the leftmost wall, gradually integrating the depth maps of subsequent walls. When a new wall is integrated, an intersection line is formed between the new plane depth map and the existing plane depth map. In an actual room layout, the two walls should be located on both sides of this intersection line. The position of each wall relative to the intersection point is determined with reference to the centroid position derived from the plane semantic label. The area of ​​the plane depth map that extends beyond the intersection line is trimmed, resulting in an updated layout depth map. This process is applied repeatedly until the plane depth maps of all walls are integrated.

[0095] Step 4.3, through the fine integration process, a complete and accurate 3D layout depth map is obtained. Finally, the depth maps of the ceiling and floor are integrated into the layout depth map using a pixel minimum operation. From the obtained layout depth map, a layout segmentation mask can be generated based on the plane category associated with each depth value at the same time.

[0096] To further enhance the robustness of the prediction, during the test phase, the input image is horizontally flipped and the alternative layout estimation result (horizontally flipped) is obtained in the same way. For the original image and the flipped image, the pixel-level matching rate between each layout segmentation mask obtained by sequential integration and S is evaluated, and the one with the higher matching rate is finally selected as the final layout estimation. This method ensures the reliability and accuracy of the prediction.

[0097] Example

[0098] Implementation of perspective images. The perspective image network is built on a relatively simple architecture designed for perspective depth estimation, which uses SENET-154 as the encoder. The output branch for full depth map prediction includes a decoder, a multi-scale feature fusion module, and a refinement module. The resolution of the input image is set to 256x256, and the resolution of the output depth map is 128×128. The output branch is also extended with a sigmoid activation for predicting the global plane visibility confidence map and the plane semantic label. The output branch for plane category prediction starts with a 1×1 convolutional layer, which also reduces the channel dimension to 256. After this, a bilinear interpolation layer resizes the spatial resolution of the feature map to M×M. Subsequently, 7 consecutive layers of 3×3 convolutions are applied, with batch normalization and the REU activation function combined between each pair of consecutive convolutional layers. The REU activation function is an activation function for deep learning that aims to address some limitations of the traditional ReLU activation function and is used in some deep learning models, especially when dealing with complex datasets or when higher model expressiveness is required. S-type activation is applied at the output layer. The size of the flat class prediction is M×M, where M is set to 16 by default.

[0099] The ADAMP optimizer with a batch size of 20 was also used, with an initial learning rate of 0.0001, which was decreased by a factor of 0.2% every 70 training rounds. The training images were augmented by random horizontal flipping, random rotation within ±5 degrees, and color jittering. The weighting coefficients were experimentally set to: u|=2, u2=10, u3=|, u4=20, and u5=2.

[0100] Evaluation Metrics:

[0101] The evaluation metrics for layout estimation based on perspective images include i) pixel error (PE); ii) corner error (CE); iii) 3D corner error (3DCE), which represents the average distance of the corresponding layout corner in the camera coordinate system; iv) root mean square error (Rmse); v) absolute relative error (Rel); vi) average log10 error (Log10); vii) threshold accuracy (δ1, δ2, δ3), which are δ1, δ2; 1.252, δ3; 1.253 respectively. Specifically, metrics iv-vii are used to evaluate the predicted layout depth map against the true value.

[0102] Results of pivoting the dataset

[0103] We also conduct experiments on four perspective datasets: Matterport3D-Layout, InteriorNet-Layout, LSUN, and Hedau. The dataset description and experimental results are shown below.

[0104] Matterport3D-Layout. The dataset is also constructed by careful selection and annotation of the Matterport3D dataset. The Matterport3DLayout dataset relies specifically on perspective images. It includes a training set of 4,939 images, a validation set of 456 images, and a test set of 1,965 images, including both cuboid and non-cuboid room types.

[0105] Table I: Quantitative results of overall layout estimation based on perspective images on the Matterport3D-Layout dataset:

[0106]

[0107] Specifically, PlaneAE, GeoLayout, Zhang et al., and Yao et al. are all based on direct learning of plane parameters. From the experimental results, it can be seen that our method significantly outperforms all existing methods. Compared with GeoLayout, our method reduces the errors by 1.74%, 1.34%, and 2.46% in PE, CE, and 3DCE, respectively. The metrics related to the layout depth map also show considerable performance improvements. This confirms the effectiveness of C2P transformation compared to directly learning plane parameters. In addition, from the Matterport3D dataset, all RGB-D perspective images of buildings corresponding to the training set are collected. After filtering out some unqualified images, a total of 125,419 RGB-D images are collected. The performance of the model pre-trained on this external RGBD dataset is significantly improved. Considering the richness of current indoor RGB-D perspective image datasets, this method has great potential for further improvement.

[0108] Visualization results of perspective layout estimation on Matterport3D-Layout and InteriorNet-Layout datasets, such as Figure 4 As shown, Figure 4(a) shows the visualization results of the proposed method on the Matterport3D-Layout dataset, where GeoLayout, Planeae, and Yao et al. are used for comparison. From the learned plane visibility confidence map, it can be observed that the learned confidence scores of the occluded object regions are very low, and the outlines of the object boundaries are clearly visible. Compared with other compared methods, this method has the highest accuracy and can detect planes with smaller areas, such as the tilted plane on the right side of the third column image.

[0109] InteriorNet-Layout. The dataset was created by selecting and annotating images from the photorealistic synthetic dataset InteriorNet. The training, validation, and test sets consist of 6534, 833, and 852 images, respectively. This dataset also provides comprehensive 2D and 3D layout labels, covering both cuboid and non-cuboid room types. It is worth noting that a notable feature of the InteriorNet-Layout dataset is the abundance of multi-wall samples, with 21% of the samples containing more than three walls. Compared with other datasets, the richness of multi-wall samples highlights the rich diversity of this dataset.

[0110] Table II: Quantitative results of overall layout estimation based on perspective images on the InteriorNet-Layout dataset:

[0111]

[0112] The proposed method achieves the best performance in all metrics, reducing pixel error (PE) by 3.28%, corner error (CE) by 1.48%, and 3D corner error (3DCE) by 1.69% compared to the state-of-the-art methods. These results show that the advantages of the proposed method are further amplified on datasets containing more complex multi-wall samples, demonstrating the robustness of the method in dealing with complex multi-wall scenes.

[0113] The visualization results of the InteriorNet-Layout dataset are as follows Figure 4 (b). Compared with other comparison methods, this method provides the most accurate layout estimation results, especially in multi-wall scenes. It is worth noting that compared with the Matterport3D-Layout dataset, the plane visibility confidence map learned from this dataset is sparse and mainly concentrated on the boundaries of the plane. The possible reason is that InteriorNet-Layout belongs to a synthetic dataset with relatively low ground truth depth noise. This favorable condition makes the learned plane depth closer to the plane depth equation. Therefore, C2P conversion requires fewer pixels to accurately match the plane depth parameters.

[0114] LSUNandHedau. The LSUN dataset consists of 4000 training images, 394 validation images, and 1000 test images. In this dataset, all samples belong to the cuboid room type and only provide two-dimensional layout labels. Due to the lack of depth information, this dataset is not suitable for direct training of the model. Instead, training is performed on the Matterport3D-Layout dataset. Subsequently, the RGB-D images of Matterport3D-Layout are combined to fine-tune the comprehensive depth map prediction branch, while other components of the model are refined using data from the LSUN dataset. The loss function is modified to: here The optimization was performed using the Matterport 3D-Layout dataset. and Optimization was performed using the LSUN dataset. is the cross entropy loss between the defined scaled inverse depth soft maximum and the ground truth layout segmentation.

[0115] This term encourages the plane depth value reconstructed within the corresponding plane region to be the minimum value among all plane depth maps. A slight modification is made, where the ground truth layout depth map is replaced by a depth map consisting of the minimum depth value of each pixel on all predicted plane depth maps. The weight coefficients are set to η1=1, η2=10, η3=1, η4=1, η5=10. Since the ground truth results on the LSUN test set are not publicly available, we stick to the same evaluation practice as the existing methods and test on the validation set.

[0116] The Hedau dataset consists of 209 training images and 105 test images, all of which are cuboid type. This dataset only provides ground truth layout segmentation masks. Due to its limited size, only the model trained on LSUN is used to test this dataset.

[0117] Table III: Quantitative results of cuboid layout estimation based on perspective images on LSUN and Hedau datasets:

[0118]

[0119] The performance is shown in Table III. The method achieves the lowest pixel error and corner error on the LSUN dataset and the second lowest pixel error on the Hedau dataset. The performance of the method on the 2D layout dataset is not as impressive as its performance on the 3D dataset. This difference can be attributed not only to the lack of true depth data, but also to the inconsistency between some manually annotated 2D layouts and 3D geometric constraints in the LSUN and Hedau datasets. This inconsistency becomes particularly evident in cases involving severe occlusions, where the annotators have to rely on experienced guesses to annotate the occluded parts of the layout.

[0120] Ablation

[0121] Qualitative comparison results of layout estimation and supervision plane visibility confidence, such as Figure 5 As shown, the comparison is evaluated on the MatterportLayout (top) and Matterport3DLayout (bottom) datasets. The red line is the true layout and the green line is the prediction. The main differences are highlighted with dashed lines.

[0122] Table IV: Quantitative comparison of perspective layout estimation with planar visibility confidence supervision on the Matterport3D-Layout dataset:

[0123]

[0124] In this method, plane visibility confidence is learned autonomously without direct supervision. To test the effectiveness of this learning strategy, pseudo-ground truth values ​​of plane visibility confidence are generated based on the pixel-wise similarity between the real comprehensive depth map and the layout depth map. As a comparison method, an L1 loss between the predicted global plane visibility confidence and the pseudo-ground truth is incorporated during training. Qualitative comparisons of supervised and unsupervised plane visibility confidence on the MatterportLayout and Matterport3D-Layout datasets are shown in Tables IV and V, respectively. The results show that methods without direct supervision show better performance. Figure 5 Qualitative comparisons are shown. It can be seen that although the plane visibility confidence map learned through direct supervision covers most of the visible plane regions, the final layout estimation result is not as accurate as the map obtained through autonomous learning. This shows that forcing the network to fit all visible plane regions is unnecessary. Instead, letting the network independently identify and focus on the most credible visible plane regions leads to more favorable results.

[0125] The qualitative comparison results of the complete depth map and the layout depth map are as follows: Figure 6As shown. Evaluation comparison on the MatterportLayout (above) and Matterport3D-Layout datasets, the red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines.

[0126] Table V: Quantitative comparison of perspective layout estimation using learned layout depth maps and full depth maps on the Matterport3D-Layout dataset:

[0127]

[0128] During the training process of C2P-Net, the depth information of the occluded part of the full depth map is obtained. This helps the network to have a comprehensive understanding of the overall spatial structure of the indoor scene. In order to verify the impact of learning the depth information of the occluded part, a comparative experiment was conducted in which the branch originally used to predict the full depth map was used to predict the layout depth map. Tables IV and V provide quantitative comparison results on the MatterportLayout and Matterport3D-Layout datasets, respectively. The results show that learning the full depth map helps to obtain better layout estimates, proving that learning the depth information of the occluded part is beneficial for enhancing the network's understanding of the room layout. Figure 6 Qualitative comparison results are presented. The results show that the network trained to learn the full depth map is able to obtain plane visibility confidence maps that are able to distinguish planes from obstacles to a certain extent compared to the layout depth map. The difference is more obvious on the perspective dataset Matterport3D-Layout. This shows that learning the full depth map enables the network to automatically distinguish between obstacles and planes, thus gaining a deeper understanding of the spatial structure of the scene. As a result, the final layout estimate is also more accurate.

[0129] Table VI: Quantitative comparison of perspective layout estimation for different M values ​​on the Matterport3DLayout dataset:

[0130]

[0131] In C2P-Net, the number of plane categories (i.e., M) used for perspective images is a key hyperparameter. The increase in the number of categories improves the network's ability to distinguish dense narrow planes, thereby pushing up the upper limit of the accuracy that the method can achieve. However, this expansion in categories also upgrades the challenge of accurate classification and expands the channels for learning plane semantic labels. As a result, some channels become prone to over-adaptation due to insufficient training. In Table VI, the perspective layout estimation results for different M values ​​are evaluated on the Matterport3D-Layout dataset. The model with M=16 has the best overall performance.

[0132] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A perspective view C2P conversion and layout generation method for room layout estimation, comprising: Step 1: Use a parametric method to represent the plane depth in the perspective image; according to the pinhole camera model, the projection of the 3D point (x, y, z) from the camera coordinate system to the pixel (u, v) in the perspective image is as follows: where f x , f y is the focal length, (u0, v0) is the principal point, if we change the equation to If any plane is projected onto the image, its pixel coordinates and depth values ​​will satisfy This equation can be restated as: Here, Defined as the planar depth parameter of a plane in a perspective image; (a^, b^, c^) is the normal vector, d^ is the distance to the camera center; σ, are the three parameters that define the depth of the plane; Step 2: Based on the plane depth parameterization, C2P conversion is introduced; Step 3: Design of perspective image; Classify the planes and define M×M anchor points evenly distributed on the perspective image. For each dominant plane including the ceiling and the floor, calculate the centroid coordinates of the corresponding segmented region. The label assigned to the plane corresponds to the index of the anchor point closest to the centroid. Step 4: Sequential planar depth map integration.

2. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 2 introduces C2P transformation including: the goal of C2P transformation is to derive a representation of P k The plane depth parameter of plane k is obtained, thereby facilitating plane depth reconstruction; let the plane be k, let Z represent the complete depth map of the perspective image, represents the plane visibility confidence, Q k Denotes the target plane depth parameter of plane k, constructed using the homogeneous pixel coordinates obtained from Z and their corresponding inverse depth values and The formula is as follows: To ensure consistency, di is also used to represent the inverse depth value of the i-th pixel in the perspective image, that is, Second, use the elements on the diagonal Construct a diagonal matrix The target plane depth parameter Qk is calculated as follows: For planar depth map reconstruction, construct and in Given The reconstructed depth map of plane k is calculated as follows: Z represents the comprehensive depth map of the perspective image, and the homogeneous pixel coordinates obtained from Z and their corresponding inverse depth values ​​are used to construct the X~ and Y~ matrices respectively; n is the total number of pixels in the image; u n , Represents the horizontal and vertical coordinates of the nth pixel; d n Represents the inverse depth value of the nth pixel; Zi represents the depth value of the i-th image; σ, are the three parameters that define the depth of the plane; A parameter representing the plane depth of the Kth plane; The goal of the C2P transformation is to derive plane depth parameters for plane k denoted as Pk, thereby facilitating plane depth reconstruction; the C2P transformation is differentiable for perspective images, allowing the gradients of the loss function applied to the estimated plane depth parameters and the reconstructed plane depth map to be back-propagated to the plane visibility confidence.

3. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 2, characterized in that: The plane visibility confidence is learned autonomously without direct supervision, i.e., it is unnecessary to force the network to fit all visible plane regions. Instead, letting the network independently identify and focus on the most credible visible plane regions will lead to more favorable results.

4. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 3 further comprises: obtaining a plane k whose region centroid is closest to the anchor point in the i-th row and j-th column, where k = (i-1)M+j. Through this labeling strategy, a total of M 2 different plane categories; the output branch responsible for predicting the plane category produces a probability map that is aligned with the distribution space of the anchors If there is a plane k in the image, then The i-th row and j-th column of The value in should be 1, otherwise it should be 0; The learning of is supervised by a binary map generated from the ground truth layout segmentation mask, denoted as Considering There is a significant imbalance between positive and negative samples, using and The weighted cross entropy loss between is used as the loss function: where t is the weighting factor set to t = 25 in the experiment; k represents the label of the plane; is the probability that the output branch responsible for predicting the plane category will produce an alignment with the distribution space of the anchor picture; The output branch for the true plane class produces a probability map that is aligned with the distribution space of the anchors.

5. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 4, characterized in that: The step 3 further includes: using a decoupling method, the number of output channels is increased from M 2 is reduced to 2M, where M channels are dedicated to predicting the x-axis component α∈R h×w×M , the remaining M channels are dedicated to predicting the y-axis component β∈R h×w×M , the j-th channel mapping of α is represented as α (j) (j=1, 2, ..., M), which is a probability map indicating whether the plane class of each pixel is associated with an anchor in the jth column; similarly, β (i) (i=1, 2, ..., M) indicates whether the plane category of each pixel is associated with one of the anchors in the i-th row, so α j and β i The joint probability represents the semantic label of the plane associated with the anchor in the i-th row and j-th column. The plane category is (i-1)M+j, defined as To represent M 2 All semantic labels of the plane category, since α and β are independent, Each channel map in can be obtained by element-wise multiplication of the channel maps in α and β: in Indicates the right-to-speak operation; Based on the above analysis, the corresponding output branch is used to predict the decoupled plane semantic labels of the 2M output channels. The training objective of the decoupled planar semantic label represented by is composed of the sum of the binary cross entropy loss and the dice loss specified in the formula; α (j) (j=1, 2, ..., M) is a probability map indicating whether the plane class of each pixel is associated with the anchor in the jth column; β (i) (i=1, 2, ..., M) represents the probability map of whether the plane category of each pixel is associated with the anchor in the i-th row; α j and β i represents the joint probability of the semantic labels of the plane associated with the anchor point in the i-th row and j-th column.

6. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 4 further comprises: Step 4.1, determine the wall category in the image, and use the C2P conversion technology to generate the plane depth map of the ceiling, floor and each wall; Step 4.2, distinguishing the selected plane into walls and non-walls by classifying the estimated plane depth parameters; Step 4.3: Through a fine integration process, a complete and accurate 3D layout depth map is obtained.

7. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 6, characterized in that: The step 4.1 further includes: using the plane category prediction C, defining a threshold δ c , and select the predicted value greater than δ c plane category; whenever the wall category is selected, its two neighboring categories are checked. If any of these neighboring categories is selected, the dice loss is calculated to evaluate the similarity between the predicted semantic labels of these neighboring planes. The dice loss exceeds a predefined threshold δ d Indicates that adjacent predictions correspond to different walls, encouraging both to be retained. On the contrary, when the dice loss is lower than δ d , it is suggested to keep only the wall categories with higher prediction values.

8. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 6, characterized in that: The step 4.2 further comprises: having an estimated plane depth parameter The selected plane k, the second parameter σ k is proportional to the perpendicular component of the plane normal if the ratio Below a predefined threshold δ q , classify plane k as a wall, otherwise, classify plane k as a non-wall; after distinguishing all the walls, sort them from left to right based on the horizontal coordinates of the centroids associated with the plane semantic labels, and the plane depth map is regarded as a 3D surface in the coordinate system defined by the pixel coordinates and the depth axis; Represents the plane depth parameter of the defined k-th plane; The formula is: σ k is the second parameter of the plane parameters of the k-th plane.

9. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 6, characterized in that: The step 4.2 further includes: analyzing the integral of the two plane depth maps as the intersection of the two surfaces in the 3D coordinate system, starting the process with the depth map of the leftmost wall, gradually integrating the depth maps of subsequent walls, and when integrating new walls, an intersection line is formed between the new plane depth map and the existing plane depth map; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label, trimming the area of ​​the plane depth map extending beyond the intersection line, thereby generating an updated layout depth map, and this process will be repeatedly applied until the plane depth maps of all walls are integrated.

10. The perspective view C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The perspective image network is built on top of the architecture designed for perspective depth estimation, which adopts SENET-154 as the encoder. The output branch for full depth map prediction includes a decoder, a multi-scale feature fusion module, and a refinement module.

Images