A Panoramic C2P Conversion and Layout Generation Method for Room Layout Estimation

Abstract

The present invention discloses a panoramic image C2P conversion and layout generation method for room layout estimation. It comprises: acquiring a target indoor scene image; inputting the target indoor scene image into a deep learning model, outputting a predicted complete depth map, a global plane visibility confidence map, a wall category and a plane semantic label; assigning labels according to the horizontal position of the centroid of the segmented area corresponding to each wall; determining the category associated with the highest label prediction at each pixel according to the plane semantic label, thereby obtaining a rough layout segmentation mask. Plane depth parameters are generated from the complete depth map and the plane visibility confidence. A plane depth map is generated using C2P conversion technology, and the plane layout of the room is accurately estimated by integrating sequential plane depth maps. The present invention enables C2P transformation to be flexibly applied to panoramic images, providing strong technical support for plane depth reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a panoramic image C2P conversion and layout generation method for room layout estimation. Background Art

[0002] Room layout estimation is a challenging problem in computer vision, which aims to infer the overall spatial layout of indoor environments from panoramic images. Applications include scene reconstruction, indoor localization, and augmented reality.

[0003] Room layout is determined by major indoor planes, specifically the floor, ceiling, and walls. Recovering 3D room layout essentially involves reconstructing these major indoor planes, which is a fundamental challenge in layout estimation. Some recent studies have reconstructed these major planes in images by learning plane parameters. However, a significant limitation of these methods is the lack of a mechanism to guide the network to focus on the visible regions of major indoor planes, which makes the prediction susceptible to occlusions. In addition, the determination of plane parameters is influenced not only by visual appearance but also by the positional context of the planes in the image. This dual dependency increases complexity and places an additional burden on network fitting. Summary of the Invention

[0004] The purpose of the present invention is to provide a panoramic image C2P conversion and layout generation method for room layout estimation, so as to solve the technical problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides a panoramic image C2P conversion and layout generation method for room layout estimation, comprising:

[0006] Step 1. Parameter representation of plane depth in equirectangular panoramic images; store the distance from the camera center position to each direction point, and the pixel p = (u, v) with depth value R in the equirectangular panoramic image is converted to a 3D point in the spherical coordinate system centered on the camera

[0007] Step 2: Based on the plane depth parameterization, C2P conversion is introduced;

[0008] Step 3: Reconstruct the layout of the panoramic image by predicting the depth map of each plane. According to the C2P transformation, a complete depth map is predicted, the main planes in the image are identified, and the visibility confidence of each plane is determined.

[0009] Step 4: Sequential planar depth map integration.

[0010] Further, in step 1 Where r is point P s is the radial distance to the camera center, and and θ are respectively determined and expressed as

[0011]

[0012] Where W and h are the width and height of the panoramic image in pixels, respectively, and P s Further conversion to Cartesian coordinates yields P c :

[0013]

[0014] Assuming the point lies on a plane in 3D space, the equation is By combining formula (2), the relationship between plane depth and pixel coordinates in the panoramic image can be derived as follows:

[0015] in Define P = [φψχ] T is the depth parameter of a plane in a panoramic image, which can be used to determine the depth map of any plane. Formula (3) and (1) together establish the mathematical relationship between the pixel coordinates and depth values ​​of a plane in a panoramic image, thereby allowing the plane depth to be reconstructed in a panoramic image using the plane depth parameter.

[0016] Furthermore, the C2P transformation introduced in step 2 includes: taking a plane represented by k as an example, given a complete depth map represented by R and a plane represented by V k The goal of the C2P transformation is to derive the plane depth parameters of plane k denoted as Pk, thereby facilitating plane depth reconstruction, using the pixel coordinates and their associated inverse depth values ​​derived from R, forming two matrices, X∈Rn×3 and Y∈Rn.

[0017] Furthermore, step 3 further includes: decomposing the complex visibility confidence prediction task into two subtasks, including:

[0018] Step 3.1: Predict the semantic label of each plane to identify the main planes in the scene;

[0019] Step 3.2: Repeat the last output branch to predict a global plane visibility confidence map, which intuitively reflects the visibility of all major planes.

[0020] The above decomposition into two subtasks enables each subtask to focus on its designated scope, thus reducing learning complexity.

[0021] Furthermore, step 3.1 further includes a four-task learning framework for accurate 3D layout estimation; the C2P network architecture of the panoramic image network prediction consists of a feature encoder and four independent output branches, each of which performs four different prediction tasks, including generating a complete depth map, identifying the wall type in the image, assigning semantic labels to each plane, and generating a confidence map of global plane visibility; specifically including:

[0022] i) Generate a complete depth map. For a given indoor panorama, use the first output branch to predict the complete depth map.

[0023] ii) Identifying wall types in images. A panoramic image of a room typically includes a ceiling, floor, and multiple walls. However, when labeling and identifying these elements, the definition of the ceiling and floor is relatively clear, while the definition of walls is more fuzzy and ambiguous.

[0024] iii) Assign semantic labels to each plane. For a given indoor panorama, the predicted plane semantic labels are named S∈R h×w×(N+2) , N is the number of anchor points uniformly distributed along the horizontal direction of the panoramic image, and 2 represents the ceiling and the ground;

[0025] iiii) generating a global plane visibility confidence map, the panoramic image network can reconstruct a plane depth map based on the plane visibility confidence map, thereby allowing the panoramic image network to autonomously determine visibility confidence according to actual needs.

[0026] Furthermore, the C2P network architecture in the panoramic image network prediction is based on the monocular depth estimation network. The C2P network architecture of the panoramic image uses SwinTransform as the encoder, the pyramid module to aggregate information, and the neural window FC-CRFS module as the decoder to predict the complete depth map.

[0027] Furthermore, the step 4 further comprises:

[0028] Step 4.1: Determine the wall categories in the image and generate planar depth maps of the ceiling, floor, and each wall using C2P conversion technology.

[0029] Step 4.2, sort the walls from left to right and integrate their respective planar depth maps in a sequential, step-by-step manner;

[0030] Step 4.3: A complete and accurate 3D layout depth map is obtained through a fine integration process, where the depth maps of the ceiling and floor are integrated into the layout depth map using a pixel-by-pixel minimization operation.

[0031] Furthermore, step 4.2 further includes: analyzing the integration of two planar depth maps in a three-dimensional coordinate system, starting the process with the depth map of the leftmost wall as a result of the intersection of two surfaces, and gradually integrating the depth maps of subsequent walls; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label; trimming the area of ​​the planar depth map that extends beyond the intersection line, thereby generating an updated layout depth map; this process is repeatedly applied until the planar depth maps of all walls are integrated.

[0032] Furthermore, by learning a comprehensive and complete depth map including planes and objects, the network can automatically distinguish obstacles from planes to fully and deeply understand the overall spatial structure of indoor scenes. At the same time, the joint learning of plane visibility confidence gives the network the ability to detect occlusions and focus on the visible area of ​​the dominant plane.

[0033] Furthermore, the network prediction does not depend on the specific location, directly solves the key problem of plane parameter learning, and can fully utilize the rich external RGB-D dataset to enhance the network prediction performance, further improving the practicality and generalization ability of the network prediction.

[0034] The method of the present invention has the following advantages:

[0035] 1) This paper introduces the C2P transform, a differentiable operation specifically designed for robust planar depth reconstruction. This method, based on a derived parametric representation of planar depth, enables the C2P transform to be flexibly applied to panoramic images, providing powerful technical support for planar depth reconstruction.

[0036] 2) This paper proposes an efficient and versatile four-task 3D layout estimation learning framework that is highly applicable to panoramic images. Based on the network's predictions, we further propose an innovative sequential planar depth map integration method to generate accurate 3D layouts. This approach not only improves the accuracy and reliability of 3D layout estimation but also demonstrates the framework's potential for processing diverse image types.

[0037] 3) The method proposed in this paper achieves state-of-the-art performance in terms of layout estimation benchmarks for panoramic images.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the C2P network for panoramic images;

[0040] Figure 2 Schematic diagram for eliminating duplicate predictions of the wall category based on the similarity of corresponding semantic labels;

[0041] Figure 3 A diagram showing the visualization results of panoramic layout estimation on the MatterportLayout dataset;

[0042] Figure 4 This is a schematic diagram of the qualitative comparison results of the MatterportLayout dataset;

[0043] Figure 5 Schematic diagram of the qualitative comparison results of the Zind dataset;

[0044] Figure 6 Schematic diagram of the qualitative comparison results of layout estimation and supervised plane visibility confidence;

[0045] Figure 7 Schematic diagram of the qualitative comparison results of learning the complete depth map and the layout depth map. DETAILED DESCRIPTION

[0046] The technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments. However, it should be understood by those skilled in the art that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0047] This invention provides an innovative room layout estimation method, the core of which is plane depth reconstruction. First, a parametric representation of plane depth in equirectangular panoramic images is derived. Subsequently, based on this foundation, a complete depth map to plane depth (C2P) conversion method is proposed. This method can be effectively applied to panoramic images. Through C2P conversion, the plane depth of the wall outlined in the image is reconstructed using the complete depth map and the plane visibility confidence map. By utilizing the parametric representation of plane depth, the C2P conversion method transforms the plane depth reconstruction problem into the prediction of complete depth and plane visibility confidence. Specifically, the complete depth map captures the depth information of visible plane regions and occluding objects, while the plane visibility confidence reflects the probability that the plane is observed at each pixel position, that is, it is not occluded by indoor objects or other salient planes. After obtaining the predicted complete depth map and plane visibility confidence, a weighted least squares method is used to solve the problem. This solution is based on the premise that the reconstructed plane depth should be closely aligned with the complete depth map in areas with high plane visibility confidence. This method can accurately estimate the room's floor plan. The detailed implementation steps and principles of C2P conversion of panoramic images will be described in detail later.

[0048] Building on the C2P transformation approach, this paper further proposes an efficient four-task learning framework for accurate 3D layout estimation. This framework employs two distinct architectures tailored for panoramic images, aiming to simultaneously learn comprehensive depth maps and plane visibility confidence. Due to the differentiability of the C2P transformation, necessary constraints are imposed on both the predicted plane depth parameters and the reconstructed plane depth map, enabling the network to autonomously learn and optimize plane visibility confidence as needed. During the testing phase, an innovative sequential plane depth map integration method, applicable to panoramic images, is introduced to generate high-quality 3D layouts. By learning a comprehensive depth map encompassing both planes and objects, the network is able to fully understand the overall spatial structure of indoor scenes. Simultaneously, the joint learning of plane visibility confidences empowers the network to detect occlusions and focus on the visible regions of dominant planes. Compared to traditional methods that directly learn plane parameters, this approach demonstrates superior robustness to occlusions. Furthermore, the network's predictions are independent of specific location, directly addressing the key issue of plane parameter learning. Although extending labeled 3D layout datasets is costly, our method is able to fully leverage abundant external RGB-D datasets to enhance its performance, further improving its practicality and generalization ability.

[0049] Panoramic room layout estimation

[0050] Many studies have used convolutional neural networks to learn 2D layout representations from panoramas and adopted the Manhattan world assumption in post-processing. By assuming that intersecting walls are vertical, a proportional room layout can be reconstructed from the predicted 2D layout representation. In addition, the absolute scale can be determined by the camera height constraint. Zou et al. used an encoder-decoder network to learn probability maps of boundaries and corners from indoor panoramas and incorporated 3D layout parameter regression to enhance learning. Sun et al. exploited the prior alignment of wall boundaries to be vertical and proposed HorizonNet, which simplifies the 2D density prediction of intersecting boundaries into a compact one-dimensional regression. The architecture of HorizonNet was refined in the subsequent work HoHoNet. Tabata et al. proposed to exploit knowledge distillation by training a teacher network on panoramas and 3D shapes. The trained teacher network then reinforced the student network to improve the prediction of ceiling-wall and floor-wall boundaries.

[0051] In addition to learning directly from panoramic images, some studies have used equirectangular-to-perspective (E2P) transformation to convert panoramic images into perspective images, such as ceiling view images. Yang et al. proposed jointly learning a ground-ceiling probability map from the panorama and a ground floor map from the ceiling map. Pintore et al. also employed E2P transformation to learn ceiling and ground boundary probability maps from the ceiling and ground views, respectively. Zhao et al. converted a panorama into six stereo images and predicted the cross-boundary within each stereo image using the Manhattan Hough transform. Wang et al. focused on complex rooms and, using a pair of panoramas as input, introduced a cross-perspective projection layer that produces a perspective image as an additional input for learning the floor map. Researchers have begun to delve into the depth information in panoramas, thereby enhancing layout learning and reasoning by leveraging 3D reasoning. Zeng et al. introduced a method for reconstructing 3D layouts by learning a layout depth map and combining it with coarse depth estimation and semantic prediction. Wang et al. proposed LED2Net, which features a differentiable depth representation mechanism that reformulates the layout learning objective as horizontal depth prediction. Jiang et al. proposed LGT-Net, which utilizes an omnidirectional geometry-aware loss to learn both horizontal depth and room height for layout representation. Jing et al. further extended LGT-Net to address the data imbalance problem. However, this method currently does not address the problem of parametric representation of planar depth in panoramas, which can enhance the research of planar depth reconstruction and layout estimation.

[0052] Planar depth reconstruction formula

[0053] The present invention provides a C2P (full depth map to plane depth) conversion method for panoramic images. The core of this method is to generate plane depth parameters from the full depth map and the plane visibility confidence. The plane visibility confidence here is a key indicator, which represents the probability that the plane is visible at each pixel position, that is, the probability that the position is not blocked by indoor objects or other major planes. In indoor scenes, areas with high visibility confidence are often parts where there are no obstacles and planar features can be clearly observed. Based on this understanding, it is expected that the plane depth map reconstructed using the estimated plane depth parameters can maintain a close alignment with the full depth map of these areas. To achieve this goal, an optimization strategy is adopted, namely, determining the depth parameters of each plane by minimizing the weighted distance between the reconstructed plane depth map and the full depth map. Here, the plane visibility confidence plays a key role as an important weighting factor. Specifically, when the visibility confidence of a certain area is high, it is more inclined to make the reconstructed plane depth map consistent with the full depth map of the area to ensure the accuracy of the estimated plane depth parameters. In this way, the C2P conversion method proposed not only considers the geometric characteristics of the plane, but also makes full use of the visibility information of the plane, thereby achieving robust estimation of the plane depth of indoor scenes.

[0054] C2P transformation of panoramic images:

[0055] Step 1. Parameter representation of plane depth in equirectangular panoramic images. The depth map of an equirectangular panoramic image is different from the standard perspective depth map. Instead of representing the distance along the z-axis (forward direction) of the camera coordinate system, it stores the distance from the camera center position to each direction point. In this case, the pixel p = (u, v) with a depth value R in the equirectangular panoramic image can be converted to a 3D point in a spherical coordinate system centered on the camera. Here, r is point P s is the radial distance to the camera center, and and θ are respectively determined and expressed as

[0056]

[0057] Where w and h are the width and height of the panoramic image in pixels respectively. Then P s Further conversion to Cartesian coordinates yields P c :

[0058]

[0059] Assuming the point lies on a plane in 3D space, the equation is By combining Formula 2, the relationship between plane depth and pixel coordinates in a panoramic image can be derived as follows:

[0060]

[0061] in Define P = [φψχ] T is the depth parameter of a plane in a panoramic image, which can be used to determine the depth map of any plane. Formula (3) and (1) together establish the mathematical relationship between the pixel coordinates and depth values ​​of a plane in a panoramic image, allowing the plane depth to be reconstructed in a panoramic image using the plane depth parameter.

[0062] Step 2: Based on the plane depth parameterization, the C2P transformation is introduced. Take a plane denoted as k as an example. Given the complete depth map denoted as R and the corresponding visibility confidence map of plane k denoted as Vk, the goal of the C2P transformation is to derive the visibility confidence map denoted as P k Using the pixel coordinates derived from R and their associated inverse depth values, we form two matrices, X∈R n ×3 and Y∈R n , defined as follows:

[0063]

[0064] in And (u i , v i ) represents the pixel coordinates of the i-th pixel. i represents the inverse depth value of the i-th pixel. Specifically, n is the total number of pixels in the panoramic image, R i is the depth value of the i-th pixel.

[0065] Then, construct a diagonal matrix W k , whose elements come from the diagonal V k , and W k The rows in align with the rows in X and Y:

[0066]

[0067] Where W i Indicates that the i-th pixel is in V k The visibility confidence value in .

[0068] According to formula (3), using P k The reconstructed plane depth, after vectorization and inverse operation, can be expressed as XP k As mentioned earlier, XP k It should be aligned with Y at the pixel location where the plane visibility confidence value is high. Therefore, P can be estimated by minimizing the weighted mean square error k :

[0069]

[0070] This optimization problem has a closed-form weighted least squares solution.

[0071] P k =(X T W k X) -1 X T W k Y (7)

[0072] It should be noted that there are many visibility confidence values ​​close to zero, which is not good for P k The contribution to the solution of W is very small. In order to reduce the computation and storage costs, k Find the value less than the threshold δ w The diagonal elements of , and the corresponding rows and columns are deleted. At the same time, the corresponding rows in X and Y are also deleted. w The value of is empirically set to 0.1.

[0073] Once you get P k =[φ k ψ k χ k ] T , we can use the formula to calculate the reconstructed depth map of plane k, expressed as R k To improve computational efficiency, the pixel coordinates in the panorama are manipulated to construct three different h×w images, labeled A, B, and C. Specifically, the values ​​of A, B, and C at the (u, v) coordinates are defined as and Respectively (such as Figure 2 (a) Adjacent wall class predictions correspond to different walls. (b) Adjacent wall class predictions are repeated, where only the class with the higher activation value is retained. Therefore, R is calculated as follows k :

[0074]

[0075] Among them, 1∈R h×w and 0∈R h×w is a matrix whose elements are all equal to 1 and 0 respectively. and max represent element-wise division and maximum operations, respectively.

[0076] Panoramic image frame:

[0077] Step 3: Reconstruct the layout of each plane in the panoramic image by predicting a depth map for each plane. Based on the C2P transformation, a complete depth map needs to be predicted, the main planes in the image need to be identified, and the visibility confidence of each plane needs to be determined. The complex visibility confidence prediction task is decomposed into two subtasks: predicting a semantic label for each plane to identify the main planes in the scene; second, generating a global plane visibility confidence map that intuitively reflects the visibility of all major planes. This decomposition enables each subtask to focus on its designated scope, thereby reducing learning complexity.

[0078] Specifically, the two subtasks of decomposition include the following sub-steps:

[0079] Step 3.1: Predict the semantic label of each plane to identify the main planes in the scene;

[0080] The C2P network of panoramic images is as follows Figure 1 As shown in Figure 2, the architecture consists of a feature encoder and four output branches, which are used to predict a full depth map, a global plane visibility confidence map, a wall category, and a plane semantic label. Based on the predicted wall category, the corresponding plane semantic label is extracted, allowing the derivation of a visibility confidence map for each plane. Subsequently, a depth map of the plane is estimated through a C2P transformation, and this depth map is used to generate a coarse layout depth map through weighted averaging. The entire framework can be trained in an end-to-end manner.

[0081] Figure 2 This paper outlines a network architecture designed specifically for panoramic image processing. The architecture consists of a feature encoder and four independent output branches. These branches focus on different prediction tasks, including generating a complete depth map, identifying wall types in the image, assigning semantic labels to each plane, and generating a confidence map for global plane visibility. The following is a detailed explanation of the learning details of each component:

[0082] i) Complete depth map. For a given indoor panorama, the first output branch is used to predict the complete depth map. The learning objective is defined as the sum of the following two terms:

[0083]

[0084] The first term is the proposed scale-invariant logarithmic loss. Let Denotes the difference between the predicted depth map R and the ground truth R at pixel i * The logarithmic difference between them, R* represents the true depth value, and the loss function is defined as follows:

[0085]

[0086] Here, λ is a tuning parameter that balances the importance of absolute scale error and scale invariance error. In the experiment, λ is set to 0.85.

[0087] It is expected that the predicted full depth map changes more smoothly within each plane area, while showing clear and sharp edges on the intersection line of the two main planes. To this end, the multi-scale gradient matching loss function defined as follows is adopted:

[0088]

[0089] Four scales are used, and ρ∈{1, 2, 3, 4} is used to represent the scale. Gρ refers to the depth difference map of scale ρ, that is, G 1 =RR * , G ρ The resolution is G ρ-1 Half of n ρ is the number of pixels of scale ρ.

[0090] ii) Wall Classification. Panoramic images of a room typically include a ceiling, floor, and multiple walls. However, when labeling and identifying these elements, the ceiling and floor are relatively well-defined, while the definition of walls is more fuzzy and ambiguous. In particular, when dealing with the labeling of walls, some research methods rely on empirically assigned labels, such as simply marking them as the front wall, left wall, or right wall. Although simple, this method is inherently ambiguous, resulting in inconsistent label assignments and a lack of intuitiveness and clear boundary demarcation. This inconsistency and unintuitiveness pose a challenge to the learning process, as it can make it difficult for the model to accurately identify and understand the roles and relationships of different walls in the scene. To improve this situation, it is necessary to explore more accurate, consistent, and intuitive wall labeling methods to support more precise and efficient panoramic image processing and analysis.

[0091] In a panoramic image, the ceiling and floor occupy the top and bottom regions, respectively, while walls lie horizontally between them. To accurately and consistently assign labels to walls, labels are assigned based on the horizontal position of the centroid of each segmented region. First, N anchor points are defined, evenly distributed horizontally along the panoramic image. These anchor points serve as reference points for determining the horizontal position of each wall. Next, for each wall in the image, the centroid coordinates of its corresponding segmented region are calculated. The centroid represents the horizontal center of the region and is therefore crucial for determining the wall's position. The centroid coordinates of each wall are then compared with the horizontal positions of all anchor points to identify the anchor point closest to the centroid. This closest anchor point serves as the horizontal anchor point for that wall. Finally, the index of the anchor point is used as a label and assigned to the corresponding wall. In this way, each wall is assigned a unique label that reflects its relative horizontal position within the panoramic image. This approach is both accurate and intuitive, effectively resolving ambiguity in wall labeling. It ensures the consistency of annotation, improves the accuracy of annotation, and provides strong support for subsequent panoramic image processing and analysis.

[0092] An independent output branch is used to predict the probability of each wall category in the image, denoted as C∈R N The learning of C is determined by the true value C * ∈B N , B={0,1} for supervision, where C * is generated from the ground-truth segmentation mask. Cross entropy loss is used as the training objective:

[0093]

[0094] iii) Plane semantic labels. Assign semantic labels to each plane. For a given indoor panorama, the predicted plane semantic labels are named S∈R h×w×(N+2) , N is the number of anchor points uniformly distributed horizontally along the panoramic image, 2 represents the ceiling and the floor, and the N+2 output channels correspond to the semantic labels of N types of walls, ceilings, and floors, respectively. Under the guidance of the ground truth S*, the loss function of the plane semantic label is defined as follows:

[0095]

[0096] Among them C * =[C * 11]∈B N+2 It is C * Expansion. (i) and S *(i) Represents S and S respectively * The i-th channel. L bcerepresents the binary cross entropy loss, and L dice represents the dice loss, which is defined as:

[0097]

[0098] Here, S (i) j and S *(i) j They refer to S (i) and S *(i) For the wall category missing in the image, no constraints are imposed on the corresponding plane semantic label prediction.

[0099] Given the predicted plane semantic labels, in order to provide a reference in the subsequent layout generation stage, the category associated with the highest label prediction is determined at each pixel, resulting in a coarse layout segmentation mask, denoted as S:

[0100]

[0101] It's important to note that when processing panoramic images, the leftmost and rightmost walls often represent different parts of the same wall. When predicting the wall category and corresponding plane semantic labels, since these two walls appear as separate regions in the image, their respective categories and plane semantic labels should be predicted separately. To improve prediction accuracy and completeness, the plane semantic labels corresponding to the leftmost and rightmost walls are merged during testing. This step yields a complete semantic mask covering the entire wall. This mask not only reflects the wall's category information but also its complete shape and position. Next, this complete semantic mask is used to determine the wall's plane visibility confidence. By evaluating the visibility of each pixel in the mask and incorporating information about the wall's position and angle in the panoramic image, the plane visibility of the entire wall can be more accurately assessed. This approach ensures a more comprehensive consideration of the wall's integrity and continuity when processing panoramic images, thereby improving the accuracy of wall category prediction, plane semantic label assignment, and plane visibility assessment. This network's predictions are independent of specific location, directly addressing the key issue of plane parameter learning.

[0102] iiii) Global plane visibility confidence map. In panoramic image processing, global plane visibility confidence maps often do not have direct ground truth values ​​for reference. Although obtaining all visible regions of all major planes in the training image is not particularly difficult, this can be achieved by identifying regions where the depth values ​​are closely aligned between the ground truth depth map and the layout depth map, or by manual annotation, this method focuses on optimizing this process through indirect means. As implied by formula (6), it is not required that the network must include all visible pixels to solve the problem. Instead, it is preferred that the network can assign high confidence values ​​to the most certain visible plane regions to produce robust plane depth parameter estimates. This strategy helps avoid being forced to fit all visible plane regions under strict supervision constraints, which may introduce unnecessary errors. Since the C2P transform is differentiable, this method cleverly exploits this to enhance the estimated plane depth parameters through indirect constraints. Specifically, the network is able to reconstruct the plane depth map based on the plane visibility confidence map, allowing the network to autonomously determine the visibility confidence according to actual needs. This approach not only improves the flexibility and adaptability of the system, but also helps to generate more accurate and reliable plane depth parameter estimates.

[0103] The last output branch is used to predict the global plane visibility confidence map, denoted as V∈R h×w , then the visibility confidence of each plane is determined by performing element-wise multiplication of V with the corresponding plane semantic label. Specifically, for plane k, the visibility confidence V k The calculation is as follows:

[0104]

[0105] in Represents element-by-element multiplication. For a training image, the predicted full depth map and visibility confidence map of the main plane are used to generate the corresponding plane depth parameters and plane depth map through C2P transformation.

[0106] In order to constrain the estimation of plane depth parameters, a weighted L1 loss function is used between the estimated plane depth parameters and the true value. Considering that the plane depth parameters of larger plane areas have greater importance on the layout estimation results, the weight of the loss function is adjusted according to the number of pixels in the layout segmentation mask of each plane:

[0107]

[0108] Among them, P * irefers to the true plane depth parameter of plane i. The error of the plane depth parameter alone is not enough to fully reflect the layout error. As a supplement, the reconstructed plane depth maps are weighted averaged using their corresponding plane semantic label predictions to obtain a rough estimate of the layout depth map, denoted as R:

[0109]

[0110] The loss between R and the true layout depth map is denoted as R * , is used as an additional learning objective. Consistent with formula (14). Using the same loss function, designated as L R , to measure R and R * Divergence between.

[0111] In summary, the overall loss function formula for panoramic layout learning is as follows:

[0112]

[0113] Where θ1, θ2, θ3, θ4 and θ5 are weighting coefficients.

[0114] Step 4: Sequential planar depth map integration;

[0115] After the training phase is completed, a method applicable to panoramic images is proposed to reconstruct the 3D layout by integrating sequential planar depth maps.

[0116] First, the C2P conversion technology is used to generate the plane depth maps of the ceiling, floor and each wall.

[0117] The walls are then sorted from left to right, and their respective plan depth maps are integrated in a sequential, step-by-step manner. This sequential integration ensures smooth transitions and precise connections between walls, laying the foundation for an accurate 3D layout. The plan depth maps of the ceiling and floor are also taken into account during the integration process, ensuring they blend seamlessly with the wall depth maps.

[0118] Ultimately, through a meticulous integration process, a complete and accurate 3D layout depth map is obtained. Notably, this method does not rely on the assumption that walls are perpendicular to each other, making it effective in complex indoor environments that deviate from the traditional Manhattan world assumption. This feature makes the method more flexible and applicable, allowing it to handle a variety of complex indoor scenes.

[0119] The process of eliminating repeated predictions of wall categories based on the similarity of corresponding semantic labels is as follows Figure 2 As shown, (a) adjacent wall category predictions correspond to different walls. (b) adjacent wall category predictions are repeated, where only the category with the higher activation value is retained.

[0120] The sequential planar depth map integration specifically includes:

[0121] Step 4.1: Determine the wall category in the image. Use C2P conversion technology to generate plane depth maps of the ceiling, floor, and each wall. That is, use the plane category prediction C and define the threshold δ c , and select the predicted value greater than δ c However, this selection process may introduce duplication, as nearby predictions may potentially represent the same plane within the image. Figure 3 As shown in Figure 2, the predicted semantic labels of the adjacent plane categories are used to filter these duplicates. Whenever the wall category is selected, its two adjacent categories are checked. If any of these adjacent categories is also selected, the DICE loss is calculated to evaluate the similarity between the predicted semantic labels of these adjacent planes. The DICE loss exceeds a predefined threshold δ. d Indicates that adjacent predictions correspond to different walls, prompting both to be retained. On the contrary, when the dice loss is lower than δ d When , it is suggested to keep only the wall category with higher prediction value. c =0.3,δ d =0.5.

[0122] Step 4.2: Sort the walls from left to right and integrate their respective planar depth maps in a sequential, step-by-step manner. A planar depth map can be viewed as a 3D surface in a coordinate system defined by pixel coordinates and a depth axis. For clarity, the integration of two planar depth maps within the 3D coordinate system is analyzed as the result of the intersection of two surfaces. This process begins with the depth map of the leftmost wall (note that the rightmost and leftmost walls are parts of the same physical wall but are processed separately during the layout generation phase), gradually integrating the depth maps of subsequent walls. When integrating a new wall, an intersection line is formed between the new planar depth map and the existing planar depth map. In a real-world room layout, both walls should lie on either side of this intersection line. The position of each wall relative to the intersection point is determined by referencing the centroid position derived from the planar semantic labels. Regions of the planar depth map that extend beyond the intersection line are trimmed, resulting in an updated layout depth map. This process is repeated until the planar depth maps of all walls have been integrated.

[0123] Step 4.3: Through a refined integration process, a complete and accurate 3D layout depth map is obtained. Finally, the ceiling and floor depth maps are integrated into the layout depth map using a pixel-by-pixel minimization operation. From the resulting layout depth map, a layout segmentation mask can be generated based on the plane class associated with each depth value.

[0124] To further enhance the robustness of the prediction, during the test phase, the input image is horizontally flipped and an alternative layout estimate (horizontally flipped) is obtained using the same method. For both the original and flipped images, the pixel-wise matching rate between each layout segmentation mask, obtained through sequential integration, and S is evaluated, and the one with the highest matching rate is selected as the final layout estimate. This approach ensures the reliability and accuracy of the prediction.

[0125] Example

[0126] Implementation of panoramic images. The panoramic image network is built on the basis of the proposed monocular depth estimation network, which uses SwinTransform as the encoder, the pyramid module to aggregate information, and the neural window FC-CRFS module as the decoder to predict the full depth map. The input panoramic image size is 256×512. Unlike the original model, the output resolution is adjusted to 128×256. The original decoder is copied twice to create two independent output branches, which are used to predict the global plane visibility confidence map and the plane semantic label respectively. In this copying process, the output channel of the plane semantic label prediction branch is adjusted to N+2, and the Sigmoid activation function is added to the output layer of each branch. The output branch for wall category prediction first uses a 1×1 convolution layer to reduce the channel dimension of the output features from the PPM head to 256. Then, bilinear interpolation is applied to adjust the resolution of the feature map to Then, 6 consecutive layers of 3×3 convolution are applied, with batch normalization and ReLU activation between each pair. In the experiments, N is set to 16 for cuboid layout datasets (such as Stanford2D-3D and PanoContext); N is set to 64 for general layout datasets (e.g., MatterportLayout and Zind).

[0127] A batch size of 4 was used and the AdamP optimizer was used with an initial learning rate of 0.0001. During layout training, the learning rate was decreased by a factor of 0.3 every 100 epochs. The training images were augmented by random horizontal flipping, panoramic horizontal rotation, and color jittering. The weighting coefficient was determined experimentally and was set to θ 1＝2 ,θ 2＝5 ,θ 3＝5 ,θ 4＝5 and θ 5＝2 .

[0128] Evaluation Metrics

[0129] Standard evaluation metrics for panoramic layout estimation are used, including: i) 3D Intersection-over-Union (3dioU), which calculates the overlap between the predicted and true 3D room layouts; ii) 2D Intersection-over-Union (2dioU), which calculates the overlap of floor shapes; iii) Pixel Error (PE), which evaluates the pixel-level accuracy of the layout segmentation mask; iv) Corner Error (CE), which measures the L2 distance between the predicted corners and the true corners; v) Root Mean Square Error (RMSE), which represents the average size of the errors in the layout depth map; and vi) Threshold Accuracy (δ1), which represents the percentage of pixel inliers with an error less than 1.25.

[0130] Results on the Panorama dataset

[0131] We use four panoramic datasets: Stanford2D-3D and PanoContext for rectangular layouts, and MatterportLayout and Zind for general layouts to evaluate the performance of C2P-Net. Below is an introduction to each dataset and the experimental results.

[0132] Stanford2D-3D. This dataset is obtained by annotating the layout based on the Stanford2D3DS dataset. It contains 552 panoramas, all of which show a cubic layout. The provided panorama depth map may contain some missing depth values. To address this issue, a colorization method is used to fill these missing depth values ​​before the training process. This process is also applied to the panoramic dataset MatterportLayout. The same training / validation / test split is followed. It is worth noting that some previous studies combined the training set of the Stanford 2D-3D dataset and the training set of the entire PanoContext dataset for training. However, due to the lack of complete depth maps in the PanoContext dataset, only the training set of the Stanford 2D-3D dataset is used for training.

[0133] Table I: Quantitative results of cuboid layout estimation based on panoramic images on the Stanford 2D-3D dataset:

[0134]

[0135] Table I shows the quantitative comparison results with the state-of-the-art methods. It can be seen that the proposed method (labeled as "C2P-Net") outperforms all other methods even with less training data. In addition, the potential of incorporating external RGB-D data to enhance layout estimation is explored. A total of 1,300 RGB-D panoramic images were collected from the original Stanford2D3DS dataset, excluding images in the Stanford2D3DS test set. Using this large dataset, the model was first pre-trained to learn the complete depth map and then fine-tuned on the training set of the Stanford2D3DS dataset. These results are labeled as "C2P-Net [with pre-training]". These results are encouraging: the pre-trained model further improves the performance, demonstrating the effectiveness of the method in utilizing external RGB-D data to improve the complete depth map estimation, thereby improving the accuracy of layout estimation. The network prediction is able to fully utilize the rich external RGB-D dataset to enhance the network prediction performance, further improving the practicality and generalization ability of the network prediction.

[0136] Table II: Quantitative results of cuboid layout estimation based on panoramic images on the PanoContext dataset:

[0137]

[0138] PanoContext. This dataset is also a small-scale cuboid layout dataset containing 514 panoramas. No real-world full depth maps are provided in PanoContext, which poses a challenge for training on this dataset. To address this limitation, a full depth map prediction branch is adopted to learn the layout depth map while keeping the other components of the framework unchanged. The training / validation / test split of previous methods is also followed. During training, unlike previous studies that combined the entire stanford-2D3DS dataset, only the training split of PanoContext is used. The results in Table II show that this method outperforms the existing methods on all evaluated metrics. This shows that even without access to the full depth map, this method is able to effectively learn from the layout depth map as a viable alternative.

[0139] Table III: Quantitative results of overall layout estimation based on panoramic images on the MatterportLayout dataset:

[0140]

[0141] MatterportLayout. This dataset contains 2,295 panoramic images carefully selected and annotated from the original Matterport3D dataset. It covers a wide range of room types, including room layouts with cuboid shapes and non-cuboid shapes (e.g., L-shaped, T-shaped rooms). The official train / validation / test split is adopted for evaluation. Quantitative results are shown in Table III. The proposed method achieves excellent performance on all metrics. Compared with the previous state-of-the-art method, the proposed method significantly improves by 2.69% on 2dioU and 2.08% on 3dioU. In addition, the Matterport3D dataset is utilized and 5784 RGB-D panoramic images are collected from the buildings corresponding to the training set. Evaluation of the model pre-trained with these external RGB-D panoramic images shows that the performance is further improved.

[0142] The visualization results of panoramic layout estimation on the MatterportLayout dataset are as follows Figure 3 As shown, the intermediate outputs of the prediction are given, including a comprehensive depth map, a global plane visibility confidence, and a coarse layout segmentation derived from the predicted plane semantic labels. It is worth noting that in the MatterportLayout panoramic dataset, the learned plane visibility confidence tends to be concentrated in the boundary areas and occlusions between planes or between planes, while the plane visibility confidence values ​​are generally lower in the center area of ​​the plane. This trend may be attributed to the fact that the C2P transformation shows a higher sensitivity to depth errors in the center area of ​​the plane compared to the edge area. The layout estimation results obtained by this method are very close to the actual situation, even for more complex non-cuboid room layouts, as shown in columns 2-4 of the figure.

[0143] The qualitative comparison results of the MatterportLayout dataset are as follows Figure 4 Figure 2 compares the proposed C2P-Net with LGT-Net and DOPNet. The room layout boundary is shown on the left, and the floor plan is shown on the right. The red line represents the ground truth, and the green line represents the predicted value. The horizon depth is also shown below each panoramic image. The main differences are highlighted with a dotted line.

[0144] Figure 5 Qualitative comparisons with LGT-Net and DOPNet are presented. The results in the first column show that, compared to LGT-Net and DOPNet, our method is not affected by obstacles on the right side of the panorama and consistently provides accurate predictions. The results in the second and third columns show that our method maintains its accuracy in complex scenes containing narrow walls in the panorama.

[0145] The qualitative comparison results of the Zind dataset are as follows Figure 5We present Zind, a large-scale panoramic dataset with layout annotations, providing a wide variety of room layout types, including Manhattan and non-Manhattan configurations. We experiment with the "easy" subset, which contains samples that allow layout recovery from a single panoramic image. We employ a formal training / validation / test split, with 24,882 panoramas for training, 3,080 for validation, and 3,170 for testing. Since complete depth maps are not available in the Zind dataset, we employ the same strategy implemented on the PanoContext dataset to overcome this limitation.

[0146] Table IV: Quantitative results of overall layout estimation based on panoramic images on the Zind dataset

[0147]

[0148] Table IV presents the quantitative comparison results on the Zind dataset. Our method achieves the best performance on all metrics, demonstrating its effectiveness in general room layout estimation. Figure 5 Qualitative comparisons are shown. The proposed method consistently produces the most accurate results across all examples, demonstrating its effectiveness in handling complex environments and accurately detecting narrow walls.

[0149] Ablation

[0150] Qualitative comparison results of layout estimation and supervised plane visibility confidence, such as Figure 6 As shown, the comparison is evaluated on the MatterportLayout dataset. The red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines.

[0151] Table V: Quantitative comparison of panoptic layout estimation and planar visibility confidence supervision on the MatterportLayout dataset

[0152]

[0153] Plane visibility confidence is learned autonomously without direct supervision. To test the effectiveness of this learning strategy, pseudo-ground truth of the plane visibility confidence is generated based on the pixel-wise similarity between the true global depth map and the layout depth map. As a comparison method, an L1 loss between the predicted global plane visibility confidence and the pseudo-ground truth is incorporated into the training process. A qualitative comparison of supervised and unsupervised plane visibility confidence on the MatterportLayout and MatterportLayout datasets is shown in Table V. A quantitative comparison of perspective layout estimation from learned layout depth maps and full depth maps is shown in Table VI. Figure 6A qualitative comparison is presented. It can be seen that although the plane visibility confidence map learned through direct supervision covers most of the visible plane area, the final layout estimate is not as accurate as the map obtained through independent learning. This shows that forcing the network to fit all visible plane areas is unnecessary. Instead, allowing the network to independently identify and focus on the most confident visible plane areas leads to more favorable results.

[0154] Qualitative comparison results of the learned complete depth map and the layout depth map, such as Figure 7 As shown. The comparison is evaluated on the MatterportLayout dataset, the red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines.

[0155] Table VI: Quantitative comparison of panoptic layout estimation using learned layout depth maps and full depth maps on the MatterportLayout dataset.

[0156]

[0157] During C2P-Net training, depth information for occluded portions of the full depth map is acquired. This helps the network gain a comprehensive understanding of the overall spatial structure of indoor scenes. To validate the impact of learning depth information for occluded portions, a comparative experiment was conducted in which the branch originally used to predict the full depth map was used to predict the layout depth map. Tables V and VI provide quantitative comparison results on the MatterportLayout dataset. The results show that learning the full depth map helps achieve better layout estimates, demonstrating that learning depth information for occluded portions is beneficial for enhancing the network's understanding of room layout. Figure 7 Qualitative comparison results are presented. The results show that, compared to learning a layout depth map, a network trained to learn a full depth map is able to obtain plane visibility confidence maps that are able to distinguish planes from obstacles to a certain extent. This suggests that learning a full depth map enables the network to automatically distinguish between obstacles and planes, leading to a deeper understanding of the spatial structure of the scene. As a result, the final layout estimate is more accurate.

[0158] Table VII: Quantitative comparison of panoptic layout estimation for different values ​​of N on the MatterportLayout dataset.

[0159]

[0160] In the C2P network, wall categories (i.e., N) are used for panoramic images. The increase in the number of categories improves the network's ability to distinguish dense narrow planes, thereby pushing up the upper limit of the accuracy that the method can achieve. However, this expansion in categories also upgrades the challenge of accurate classification and expands the channels for learning plane semantic labels. As a result, some channels become prone to over-adaptation due to insufficient training. In Table VII, the overall layout estimation results on the panoramic dataset MatterportLayout are compared. The results show that when N = 64, the layout estimation of all indicators is optimal.

[0161] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A panoramic image C2P conversion and layout generation method for room layout estimation, comprising: Step 1: Parameter representation of plane depth in equirectangular panoramic images; Storing the distance from the camera center position to the point in each direction, the pixel p = (u, v) with a depth value R in the equirectangular panoramic image is converted to a 3D point in the spherical coordinate system centered on the camera Step 2: Based on the plane depth parameterization, C2P conversion is introduced; Step 3: Reconstruct the layout of the panoramic image by predicting the depth map of each plane. According to the C2P transformation, a complete depth map is predicted, the main planes in the image are identified, and the visibility confidence of each plane is determined. Step 4: Sequential planar depth map integration; The C2P conversion introduced in step 2 includes: taking a plane denoted as k as an example, given a complete depth map denoted as R and a plane denoted as v k The corresponding visibility confidence map of plane k, the goal of C2P transformation is to derive P k The plane depth parameter of plane k is used to facilitate plane depth reconstruction, using the pixel coordinates derived from R and their associated inverse depth values, which are the inverse of the depth, to form two matrices, X∈R n ×3 and Y∈R n ; Step 3 further includes: decomposing the complex visibility confidence prediction task into two subtasks, including: Step 3.1: Predict the semantic label of each plane to identify the main planes in the scene; Step 3.2: Repeat the last output branch to predict a global plane visibility confidence map, which intuitively reflects the visibility of all major planes. Step 3.1 further includes a four-task learning framework for accurate 3D layout estimation. The C2P network architecture for panoramic image network prediction consists of a feature encoder and four independent output branches, each of which performs four different prediction tasks, including generating a complete depth map, identifying wall types in the image, assigning semantic labels to each plane, and generating a confidence map for global plane visibility. Specifically, the following are included: i) Generate a complete depth map. For a given indoor panorama, use the first output branch to predict the complete depth map. ii) Identifying wall types in images. A panoramic image of a room typically includes a ceiling, floor, and multiple walls. However, when labeling and identifying these elements, the definition of the ceiling and floor is relatively clear, while the definition of walls is more fuzzy and ambiguous. iii) Assign semantic labels to each plane. For a given indoor panorama, the predicted plane semantic labels are named S∈R h×w×(N+2) , N is the number of anchor points uniformly distributed along the horizontal direction of the panoramic image, and 2 represents the ceiling and the ground; iiii) generating a global plane visibility confidence map, the panoramic image network can reconstruct a plane depth map based on the plane visibility confidence map, thereby allowing the panoramic image network to autonomously determine visibility confidence according to actual needs.

2. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: In step 1 Where r is point P S is the radial distance to the camera center, and and θ are respectively determined and expressed as Where w and h are the width and height of the panoramic image in pixels, respectively. The horizontal coordinate u and vertical coordinate v of the pixel are the column and row numbers in the pixel coordinates of the image, respectively. Then P S Further conversion to Cartesian coordinates yields P c : Assuming the point lies on a plane in 3D space, the equation is is the normal vector, is the distance to the camera center; By combining formula (2), the relationship between plane depth and pixel coordinates in the panoramic image can be derived as follows: in Define P = [φψχ] T is the depth parameter of a plane in a panoramic image, which can be used to determine the depth map of any plane. Formula (3) and (1) together establish the mathematical relationship between the pixel coordinates and depth values ​​of a plane in a panoramic image, thereby allowing the plane depth to be reconstructed in a panoramic image using the plane depth parameter.

3. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The C2P network architecture in the panoramic image network prediction is based on a monocular depth estimation network. The C2P network architecture of the panoramic image uses SwinTransform as an encoder, a pyramid module to aggregate information, and a neural window FC-CRFS module as a decoder to predict the complete depth map.

4. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 4 further comprises: Step 4.1: Determine the wall categories in the image and generate planar depth maps of the ceiling, floor, and each wall using C2P conversion technology. Step 4.2, sort the walls from left to right and integrate their respective planar depth maps in a sequential, step-by-step manner; Step 4.3: A complete and accurate 3D layout depth map is obtained through a fine integration process, where the depth maps of the ceiling and floor are integrated into the layout depth map using a pixel-by-pixel minimization operation.

5. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 4, characterized in that: The step 4.2 further includes: analyzing the integration of two planar depth maps in a three-dimensional coordinate system, starting the process with the depth map of the leftmost wall as a result of the intersection of two surfaces, and gradually integrating the depth maps of subsequent walls; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label; trimming the area of ​​the planar depth map that extends beyond the intersection line, thereby generating an updated layout depth map; this process is repeatedly applied until the planar depth maps of all walls are integrated.

Images