Panorama C2P conversion and layout generation method for room layout estimation

Abstract

The invention discloses a panorama C2P conversion and layout generation method for room layout estimation. Comprising the steps of obtaining a target indoor scene image; inputting a target indoor scene image into the deep learning model, and outputting a predicted complete depth map, a global plane visibility confidence map, a wall category and a plane semantic tag; label distribution is carried out according to the centroid horizontal position of the segmentation area corresponding to each wall; a category associated with the highest tag prediction is determined on each pixel according to the planar semantic tags, thereby obtaining a coarse layout segmentation mask. A plane depth parameter is generated from the complete depth map and the plane visibility confidence. A planar depth map is generated using a C2P conversion technique, and the planar layout of a room is accurately estimated through sequential planar depth map integration. According to the method, C2P transformation can be flexibly applied to a panoramic image, and powerful technical support is provided for plane depth reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a panoramic image C2P conversion and layout generation method for room layout estimation. Background Art

[0002] Room layout estimation is a challenging problem in computer vision, which aims to infer the overall spatial layout of indoor environments from panoramic images. Applications include scene reconstruction, indoor localization, and augmented reality.

[0003] Room layout is determined by major indoor planes, especially the floor, ceiling, and walls. Recovering 3D room layout essentially involves reconstructing these major indoor planes, which is a fundamental challenge in layout estimation. Some recent studies reconstruct these major planes in images by learning plane parameters. However, a significant limitation of these methods is the lack of a mechanism to guide the network to focus on the visible regions of major indoor planes, which makes the prediction vulnerable to occlusions. In addition, the determination of plane parameters is not only affected by the visual appearance, but also by the positional context of the planes in the image. This dual dependency increases the complexity and imposes additional burden on network fitting. Summary of the invention

[0004] The purpose of the present invention is to provide a panoramic image C2P conversion and layout generation method for room layout estimation, so as to solve the technical problems existing in the prior art.

[0005] To achieve the above object, the panoramic image C2P conversion and layout generation method for room layout estimation of the present invention includes:

[0006] Step 1, parameter representation of plane depth in equirectangular panoramic images; store the distance from the camera center position to each direction point, and the pixel p = (u, v) with depth value R in the equirectangular panoramic image is converted to a 3D point in the spherical coordinate system centered on the camera

[0007] Step 2: Based on the plane depth parameterization, C2P conversion is introduced;

[0008] Step 3: Reconstruct the layout of each plane in the panoramic image by predicting its depth map. According to the C2P transformation, a complete depth map is predicted, the main planes in the image are identified, and the visibility confidence of each plane is determined.

[0009] Step 4: Sequential planar depth map integration.

[0010] Further, in step 1 Where r is point P s is the radial distance to the camera center, and and θ are determined to be expressed as

[0011]

[0012] Where W and h are the width and height of the panoramic image in pixels, respectively, and P s Further conversion to Cartesian coordinates yields P c :

[0013]

[0014] Assuming the point lies on a plane in 3D space, the equation is By combining formula (2), the relationship between the plane depth and pixel coordinates in the panoramic image can be derived as follows:

[0015] in Define P = [φψχ] T is the depth parameter of the plane in the panoramic image, which can be used to determine the depth map of any plane. Formula (3) and (1) together establish the mathematical relationship between the pixel coordinates and depth values ​​of the plane in the panoramic image, thereby allowing the plane depth to be reconstructed in the panoramic image using the plane depth parameter.

[0016] Further, the C2P transformation introduced in step 2 includes: taking a plane denoted as k as an example, given a complete depth map denoted as R and a plane denoted as V k The goal of the C2P transformation is to derive the plane depth parameters of plane k denoted as Pk, thereby facilitating plane depth reconstruction, using the pixel coordinates and their associated inverse depth values ​​derived from R, forming two matrices, X∈Rn×3 and Y∈Rn.

[0017] Furthermore, the step 3 further comprises: decomposing the complex visibility confidence prediction task into two subtasks, including:

[0018] Step 3.1: Predict the semantic label of each plane to identify the main planes in the scene;

[0019] Step 3.2, repeatedly use the last output branch to predict the global plane visibility confidence map, where the global plane visibility confidence map intuitively reflects the visibility of all main planes;

[0020] The above decomposition into two sub-steps enables each sub-task to focus on its designated scope, thus reducing learning complexity.

[0021] Furthermore, the step 3.1 further includes a four-task learning framework for accurate 3D layout estimation; the C2P network architecture of the panoramic image network prediction consists of a feature encoder and four independent output branches, and the four independent output branches each complete four different prediction tasks, and the four different prediction tasks include generating a complete depth map, identifying the wall type in the image, assigning a semantic label to each plane, and generating a confidence map label for global plane visibility; specifically including:

[0022] i) Generate a complete depth map. For a given indoor panorama, use the first output branch to predict the complete depth map;

[0023] ii) Identify the wall type in the image. A panoramic image of a room usually includes the ceiling, floor, and multiple walls. However, when labeling and identifying these elements, the definition of the ceiling and floor is relatively clear, while the definition of the wall is relatively vague and ambiguous.

[0024] iii) Assign semantic labels to each plane. For a given indoor panorama, the predicted plane semantic label is named S∈R h×w×(N+2) , N is the number of anchor points uniformly distributed along the horizontal direction of the panoramic image, and 2 represents the ceiling and the ground;

[0025] iiii) generating a global plane visibility confidence map, the panoramic image network can reconstruct a plane depth map according to the plane visibility confidence map, thereby allowing the panoramic image network to autonomously determine the visibility confidence according to actual needs.

[0026] Furthermore, the C2P network architecture in the panoramic image network prediction is based on a monocular depth estimation network. The C2P network architecture of the panoramic image uses SwinTransform as an encoder, a pyramid module to aggregate information, and a neural window FC-CRFS module as a decoder to predict a complete depth map.

[0027] Furthermore, the step 4 further comprises:

[0028] Step 4.1, determine the wall category in the image, and use the C2P conversion technology to generate the plane depth map of the ceiling, floor and each wall;

[0029] Step 4.2, sort the walls from left to right and integrate their respective plan depth maps in a sequential and step-by-step manner;

[0030] Step 4.3: Through a sophisticated integration process, a complete and accurate three-dimensional layout depth map is obtained, and the depth maps of the ceiling and floor are integrated into the layout depth map by using a pixel-by-pixel minimization operation.

[0031] Further, the step 4.2 further includes: analyzing the integration of two planar depth maps in the three-dimensional coordinate system, starting the process with the depth map of the leftmost wall as a result of the intersection of two surfaces, and gradually integrating the depth maps of subsequent walls; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label; trimming the area of ​​the planar depth map that extends beyond the intersection line, thereby generating an updated layout depth map; this process is repeatedly applied until the planar depth maps of all walls are integrated.

[0032] Furthermore, by learning a comprehensive and complete depth map including planes and objects, the network can automatically distinguish obstacles from planes to fully and deeply understand the overall spatial structure of indoor scenes. At the same time, the joint learning of plane visibility confidence gives the network the ability to detect occlusions and focus on the visible area of ​​the dominant plane.

[0033] Furthermore, the network prediction does not depend on the specific location, directly solves the key problem of plane parameter learning, and can make full use of rich external RGB-D datasets to enhance the network prediction performance, further improving the practicality and generalization ability of network prediction.

[0034] The method of the present invention has the following advantages:

[0035] 1) The present invention introduces the C2P transform, which is a differentiable operation method specifically used to achieve robust planar depth reconstruction. This method is based on the derived planar depth parameter representation, which enables the C2P transform to be flexibly applied to panoramic images and provides strong technical support for planar depth reconstruction.

[0036] 2) We propose an efficient and versatile four-task 3D layout estimation learning framework that is highly applicable to panoramic images. Based on the prediction results of the network, we further propose an innovative sequential planar depth map integration method to generate accurate 3D layouts. This approach not only improves the accuracy and reliability of 3D layout estimation, but also demonstrates the strong potential of the framework in processing different types of images.

[0037] 3) The method proposed in this invention has achieved state-of-the-art performance in terms of layout estimation benchmarks for panoramic images.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of C2P network for panoramic images;

[0040] Figure 2 Schematic diagram for eliminating duplicate predictions of wall categories based on the similarity of corresponding semantic labels;

[0041] Figure 3 A diagram showing the visualization results of the panoramic layout estimation on the MatterportLayout dataset;

[0042] Figure 4 This is a schematic diagram of the qualitative comparison results of the MatterportLayout dataset;

[0043] Figure 5 It is a schematic diagram of the qualitative comparison results of the Zind data set;

[0044] Figure 6 Schematic diagram of qualitative comparison results between layout estimation and supervision plane visibility confidence;

[0045] Figure 7 Schematic diagram of the qualitative comparison results of learning the complete depth map and the layout depth map. DETAILED DESCRIPTION

[0046] The technical scheme of the present invention will be clearly and completely described below in conjunction with specific embodiments, but those skilled in the art should understand that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention provides an innovative room layout estimation method, the core of which is plane depth reconstruction. First, a parametric expression of plane depth in an equirectangular panoramic image is derived. Subsequently, based on this foundation, a complete depth map to plane depth (C2P) conversion method is proposed, which can be effectively applied to panoramic images. Through C2P conversion, the plane depth of the wall outlined in the image is reconstructed using the complete depth map of the wall and the plane visibility confidence map. By using the parametric expression of the plane depth, the C2P conversion method converts the plane depth reconstruction problem into the prediction of the complete depth and the plane visibility confidence. Specifically, the complete depth map is responsible for capturing the depth information of the visible plane area and the occluded object, while the visibility confidence of the plane reflects the probability of the plane being observed at each pixel position, that is, it is not occluded by indoor objects or other significant planes. After obtaining the predicted complete depth map and plane visibility confidence, the weighted least squares method is further used for solution. This solution process is based on a premise that the reconstructed plane depth should be closely aligned with the complete depth map in the area with high plane visibility confidence. Through this method, the plane layout of the room can be accurately estimated. The detailed implementation steps and principles of C2P conversion of panoramic images will be described in detail later.

[0048] Based on the C2P transformation method, this paper further proposes an efficient four-task learning framework for accurate 3D layout estimation. The framework tailors two distinctive architectures for panoramic images, aiming to simultaneously learn comprehensive depth maps and plane visibility confidence. Due to the differentiability of the C2P transformation, necessary constraints are imposed on the predicted plane depth parameters and the reconstructed plane depth map, which enables the network to autonomously learn and optimize the plane visibility confidence as needed. In the testing phase, an innovative sequential plane depth map integration method is introduced, which is applicable to panoramic images to generate high-quality 3D layouts. By learning a comprehensive depth map including planes and objects, the network is able to fully understand the overall spatial structure of indoor scenes. At the same time, the joint learning of plane visibility confidence gives the network the ability to detect occlusions and focus on the visible area of ​​the dominant plane. Compared with the traditional method of directly learning plane parameters, this method shows higher robustness in dealing with occlusions. In addition, the network prediction does not depend on the specific location, which directly solves the key problem of plane parameter learning. Although extending labeled 3D layout datasets is costly, our method is able to fully leverage rich external RGB-D datasets to enhance its performance, further improving its practicality and generalization ability.

[0049] Panoramic Room Layout Estimation

[0050] Many studies have used convolutional neural networks to learn 2D layout representations from panoramas and adopted the Manhattan world assumption in post-processing. By assuming that intersecting walls are vertical, a proportional room layout can be reconstructed from the predicted 2D layout representation. In addition, the absolute scale can be determined by the camera height constraint. Zou et al. used an encoder-decoder network to learn probability maps of boundaries and corners from indoor panoramas and added 3D layout parameter regression to enhance learning. Sun et al. used the prior alignment of verticality of wall boundaries to propose HorizonNet, which simplifies the 2D density prediction of intersecting boundaries into a compact 1D regression. The architecture of HorizonNet was refined in the follow-up work HoHoNet. Tabata et al. proposed to exploit knowledge distillation by training a teacher network on panoramas and 3D shapes. The trained teacher network then reinforced the student network to improve the prediction of ceiling-wall and floor-wall boundaries.

[0051] In addition to learning directly from panoramic images, some studies have used equirectangular to perspective (E2P) transformation to convert panoramic images into perspective images, such as ceiling view images. Yang et al. proposed to jointly learn ground-ceiling probability maps from panoramas and learn ground floor plans from ceiling images. Pintore et al. also adopted E2P transformation to learn ceiling and ground boundary probability maps from ceiling and ground views respectively. Zhao et al. converted a panorama into 6 stereo images and predicted the cross boundaries within each stereo image through Manhattan Hough transform. Wang et al. focused on complex rooms and used a pair of panoramas as input. They introduced a cross-perspective projection layer that can produce perspective images as additional input for learning floor plans. At present, researchers have begun to delve into the depth information in panoramas to enhance layout learning and reasoning by leveraging 3D reasoning. Zeng et al. introduced a method to reconstruct 3D layouts by learning layout depth maps, combining coarse depth estimation and semantic prediction. Wang et al. proposed LED2Net, which has a differentiable depth representation mechanism that reformulates the goal of layout learning as horizontal depth prediction. Jiang et al. proposed LGT-Net, which uses an omnidirectional geometry-aware loss to learn the horizontal depth and room height of layout representation. Jing et al. further extended LGT-Net to address the data imbalance problem. However, the current method does not address the problem of parametric representation of planar depth in panoramas, which can enhance the research of planar depth reconstruction and layout estimation.

[0052] Planar Depth Reconstruction Formula

[0053] The present invention provides a C2P (complete depth map to plane depth) conversion method for a panoramic image. The core of the method is to generate plane depth parameters from a complete depth map and plane visibility confidence. The plane visibility confidence here is a key indicator, which represents the probability that the plane is visible at each pixel position, that is, the probability that the position is not blocked by indoor objects or other main planes. In indoor scenes, areas with high visibility confidence are often parts where there are no obstacles and plane features can be clearly observed. Based on this understanding, it is expected that the plane depth map reconstructed using the estimated plane depth parameters can maintain a close alignment relationship with the complete depth map of these areas. To achieve this goal, an optimization strategy is adopted, that is, to determine the depth parameters of each plane by minimizing the weighted distance between the reconstructed plane depth map and the complete depth map. Here, the plane visibility confidence plays a key role as an important weighting factor. Specifically, when the visibility confidence of a certain area is high, it is more inclined to keep the reconstructed plane depth map consistent with the complete depth map of the area to ensure the accuracy of the estimated plane depth parameters. In this way, the C2P conversion method proposed not only considers the geometric characteristics of the plane, but also makes full use of the visibility information of the plane, thereby achieving a robust estimation of the plane depth of indoor scenes.

[0054] C2P transformation of panoramic images:

[0055] Step 1. Parameter representation of planar depth in equirectangular panoramic images. The depth map of an equirectangular panoramic image is different from a standard perspective depth map. Instead of representing the distance along the z-axis (forward direction) of the camera coordinate system, it stores the distance from the camera center position to each direction point. In this case, a pixel p = (u, v) with a depth value R in an equirectangular panoramic image can be converted to a 3D point in a spherical coordinate system centered on the camera. Here, r is point P s is the radial distance to the camera center, and and θ are determined to be expressed as

[0056]

[0057] Where w and h are the width and height of the panoramic image in pixels respectively. Then P s Further conversion to Cartesian coordinates yields P c :

[0058]

[0059] Assuming the point lies on a plane in 3D space, the equation is By combining Formula 2, the relationship between the plane depth and the pixel coordinates in the panoramic image can be derived as follows:

[0060]

[0061] in Define P = [φψχ] T is the depth parameter of the plane in the panoramic image, which can be used to determine the depth map of any plane. Formula (3) and (1) together establish the mathematical relationship between the pixel coordinates and depth values ​​of the plane in the panoramic image, thereby allowing the plane depth to be reconstructed in the panoramic image using the plane depth parameter.

[0062] Step 2: Based on the plane depth parameterization, the C2P transformation is introduced. Take a plane denoted as k as an example. Given the complete depth map denoted as R and the corresponding visibility confidence map of plane k denoted as Vk, the goal of the C2P transformation is to derive the P k The plane depth parameters of plane k are obtained, thereby facilitating plane depth reconstruction. Using the pixel coordinates derived from R and their associated inverse depth values, two matrices are formed, X∈R n ×3 and Y∈R n , defined as follows:

[0063]

[0064] in And (u i , v i ) represents the pixel coordinates of the i-th pixel. i represents the inverse depth value of the i-th pixel. Specifically, n is the total number of pixels in the panoramic image, R i is the depth value of the i-th pixel.

[0065] Then, construct a diagonal matrix W k , whose elements come from the diagonal V k , and W k The rows in align with the rows in X and Y:

[0066]

[0067] Where W i Indicates that the i-th pixel is in V k The visibility confidence value in .

[0068] According to formula (3), using P k The reconstructed plane depth, after vectorization and inverse operation, can be expressed as XP k As mentioned earlier, XP k It should be aligned with Y at the pixel location where the plane visibility confidence value is high. Therefore, P can be estimated by minimizing the weighted mean square error k :

[0069]

[0070] This optimization problem has a closed-form weighted least squares solution.

[0071] P k =(X T W k X) -1 X T W k Y (7)

[0072] It should be noted that there are many visibility confidence values ​​close to zero, which is not good for P k The contribution of W to the solution is small. In order to reduce the computation and storage cost, k Find the value less than the threshold δ w The diagonal elements of , and the corresponding rows and columns are deleted. At the same time, the corresponding rows in X and Y are also deleted. w The value of is empirically set to 0.1.

[0073] Once you get P k =[φ k ψ k χ k ] T , we can use the formula to calculate the reconstructed depth map of plane k, expressed as R k To improve computational efficiency, the pixel coordinates in the panorama are manipulated to construct three different h×w images, labeled A, B, and C. Specifically, the values ​​of A, B, and C at the (u, v) coordinates are defined as and Respectively (such as Figure 2 (a) The adjacent wall category predictions correspond to different walls. (b) The adjacent wall category predictions are repeated, where only the category with the higher activation value is retained. Therefore, R is calculated by k :

[0074]

[0075] Among them, 1∈R h×w and 0∈R h×w is a matrix whose elements are all equal to 1 and 0 respectively. Symbol and max represent element-wise division and maximum operations, respectively.

[0076] Panoramic image frame:

[0077] Step 3: Reconstruct the layout of each plane in the panoramic image by predicting its depth map. According to the C2P transformation, a complete depth map needs to be predicted, the main planes in the image need to be identified, and the visibility confidence of each plane needs to be determined. The complex visibility confidence prediction task is decomposed into two subtasks: predicting the semantic label of each plane to identify the main planes in the scene; secondly, generating a global plane visibility confidence map, which will intuitively reflect the visibility of all main planes. This decomposition enables each subtask to focus on its designated scope, thereby reducing learning complexity.

[0078] Specifically, the two subtasks decomposed include the following substeps:

[0079] Step 3.1: Predict the semantic label of each plane to identify the main planes in the scene;

[0080] The C2P network of panoramic images is as follows Figure 1 As shown in the figure. The architecture consists of a feature encoder and four output branches for predicting the full depth map, global plane visibility confidence map, wall category, and plane semantic label. Based on the predicted wall category, the corresponding plane semantic label is extracted, allowing the visibility confidence map of each plane to be derived. Subsequently, the depth map of the plane is estimated through C2P transformation, and the depth map is used to generate a coarse layout depth map through weighted averaging. The entire framework can be trained in an end-to-end manner.

[0081] Figure 2 This paper outlines a network architecture designed for panoramic image processing. The architecture consists of a feature encoder and four independent output branches. These branches focus on different prediction tasks, including generating a complete depth map, identifying the wall type in the image, assigning semantic labels to each plane, and generating confidence labels for global plane visibility. The following is a detailed explanation of the learning details of each part:

[0082] i) Complete depth map. For a given indoor panorama, the first output branch is used to predict the complete depth map. The learning objective is defined as the sum of the following two terms:

[0083]

[0084] The first term is the proposed scale-invariant logarithmic loss. Let Denotes the predicted depth map R and the ground truth R at pixel i * The logarithmic difference between them, R* represents the true depth value, and the loss function is defined as follows:

[0085]

[0086] Among them, λ is a tuning parameter that balances the importance between absolute scale error and scale invariance error. In the experiment, λ is set to 0.85.

[0087] It is expected that the predicted full depth map changes more smoothly within each plane region, while showing clear and sharp edges on the intersection line of the two main planes. To this end, the multi-scale gradient matching loss function defined as follows is adopted:

[0088]

[0089] Four scales are used, and ρ∈{1, 2, 3, 4} is used to represent the scale. Gρ refers to the depth difference map of scale ρ, that is, G 1 =RR * , G ρ The resolution is G ρ-1 Half of n ρ is the number of pixels of scale ρ.

[0090] ii) Wall category. Panoramic images of rooms usually cover ceilings, floors, and multiple walls. However, when labeling and identifying these elements, the definitions of ceilings and floors are relatively clear, while the definition of walls is relatively vague and ambiguous. In particular, when dealing with the labeling of walls, some research methods rely on empirically assigned labels, such as simply marking them as the front wall, left wall, or right wall. Although this method is simple, its inherent ambiguity leads to inconsistency in the assignment of labels and lacks intuitiveness and clear boundary demarcation. This inconsistency and unintuitiveness poses a challenge to the learning process as it may make it difficult for the model to accurately identify and understand the roles and relationships of different walls in the scene. To improve this situation, it is necessary to explore more accurate, consistent, and intuitive wall labeling methods to support more precise and efficient panoramic image processing and analysis.

[0091] In a panoramic image, the ceiling and floor occupy the top and bottom regions of the image, respectively, while the walls are distributed horizontally between them. In order to accurately and consistently assign labels to the walls, the labels are assigned according to the horizontal position of the centroid of the segmented region corresponding to each wall. First, N anchor points uniformly distributed along the horizontal direction of the panoramic image are defined. These anchor points are used as reference points to determine the horizontal position of each wall. Next, for each wall in the image, the centroid coordinates of its corresponding segmented region are calculated. The centroid represents the center position of the region in the horizontal direction and is therefore the key to determining the position of the wall. Then, the centroid coordinates of each wall are compared with the horizontal positions of all anchor points to find the anchor point closest to the centroid in the horizontal direction. This closest anchor point is the horizontal positioning point of the wall. Finally, the index of the positioning point is used as a label and assigned to the corresponding wall. In this way, each wall is assigned a unique label that reflects the relative position of the wall in the horizontal direction of the panoramic image. This method is both accurate and intuitive, and effectively solves the ambiguity problem in wall annotation. It ensures the consistency of annotation, improves the accuracy of annotation, and provides strong support for subsequent panoramic image processing and analysis.

[0092] An independent output branch is used to predict the probability of each wall category in the image, denoted as C∈R N The learning of C is determined by the true value C * ∈B N , B = {0, 1} for supervision, where C * is generated from the ground-truth layout segmentation mask. The cross entropy loss is used as the training objective:

[0093]

[0094] iii) Plane semantic label. Assign a semantic label to each plane. For a given indoor panorama, the predicted plane semantic label is named S∈R h×w×(N+2) , N is the number of N anchor points evenly distributed along the horizontal direction of the panoramic image, 2 represents the ceiling and the ground, and the N+2 output channels correspond to the semantic labels of N types of walls, ceilings, and floors respectively. Under the guidance of the true value S*, the loss function of the plane semantic label is defined as follows:

[0095]

[0096] Among them C * =[C * 11]∈B N+2 It is C * Expansion. (i) and S *(i) Represents S and S respectively * The i-th channel. bcerepresents the binary cross entropy loss, and L dice represents the dice loss, which is defined as:

[0097]

[0098] Here, S (i) j and S *(i) j They refer to S (i) and S *(i) For the wall categories missing in the image, no constraints are imposed on the corresponding plane semantic label predictions.

[0099] Given the predicted plane semantic labels, in order to provide a reference in the subsequent layout generation stage, the category associated with the highest label prediction is determined at each pixel, resulting in a coarse layout segmentation mask, denoted as S:

[0100]

[0101] It should be noted here that when processing panoramic images, the leftmost and rightmost walls often actually represent two different parts of the same wall. When predicting the wall category and the corresponding plane semantic label, because the two walls appear as two independent regions on the image, their respective wall categories and corresponding plane semantic labels should be predicted separately. In order to improve the accuracy and completeness of the prediction, the plane semantic labels corresponding to the leftmost and rightmost walls are merged in the test. Through this step, a complete semantic mask covering the entire wall is obtained. This mask not only reflects the category information of the wall, but also contains the complete shape and position of the wall. Next, this complete semantic mask is used to determine the plane visibility confidence of the wall. By evaluating the visibility of each pixel in the mask and combining the position and angle information of the wall in the panoramic image, the plane visibility of the entire wall can be more accurately evaluated. This method ensures that the integrity and continuity of the wall can be more comprehensively considered when processing panoramic images, thereby improving the accuracy of wall category prediction, plane semantic label assignment, and plane visibility evaluation. The network prediction does not depend on the specific position, which directly solves the key problem of plane parameter learning.

[0102] iiii) Global plane visibility confidence map. In panoramic image processing, the global plane visibility confidence map often has no direct ground truth for reference. Although it is not particularly difficult to obtain all visible areas of all major planes in the training image, which can be achieved by identifying areas where the depth values ​​are closely aligned between the ground truth depth map and the layout depth map, or by manual annotation, this method focuses on optimizing this process through indirect means. As implied by formula (6), it is not required that the network must include all visible pixels to solve the problem. Instead, it is preferred that the network can assign high confidence values ​​to the most certain visible plane regions to produce robust plane depth parameter estimates. This strategy helps avoid being forced to fit all visible plane regions under strict supervision constraints, which may introduce unnecessary errors. Since the C2P transform is differentiable, this method cleverly uses this to enhance the estimated plane depth parameters through indirect constraints. Specifically, the network is able to reconstruct the plane depth map based on the plane visibility confidence map, allowing the network to autonomously determine the visibility confidence according to actual needs. This approach not only improves the flexibility and adaptability of the system, but also helps to generate more accurate and reliable plane depth parameter estimates.

[0103] The last output branch is used to predict the global plane visibility confidence map, denoted as V∈R h×w , and then determine the visibility confidence of each plane by performing an element-wise multiplication of V and the corresponding plane semantic label. Specifically, for plane k, the visibility confidence V k The calculation is as follows:

[0104]

[0105] in Represents element-by-element multiplication. For a training image, the predicted full depth map and visibility confidence map of the main plane are used to generate the corresponding plane depth parameters and plane depth map through C2P transformation.

[0106] In order to constrain the estimation of plane depth parameters, a weighted L1 loss function is used between the estimated plane depth parameters and the true values. Considering that the plane depth parameters of larger plane areas have greater importance on the layout estimation results, the weight of the loss function is adjusted according to the number of pixels in the layout segmentation mask of each plane:

[0107]

[0108] Where P * irefers to the true plane depth parameter of plane i. The error of the plane depth parameter alone is not enough to fully reflect the layout error. As a supplement, the reconstructed plane depth maps are weighted averaged using their corresponding plane semantic label predictions to obtain a rough estimate of the layout depth map, denoted as R:

[0109]

[0110] The loss between R and the true layout depth map is denoted as R * , is used as an additional learning objective. It is consistent with formula (14). The same loss function is used, designated as L R , to measure R and R * Divergence between.

[0111] In summary, the overall loss function formula for panoramic layout learning is as follows:

[0112]

[0113] Where θ1, θ2, θ3, θ4 and θ5 are weighting coefficients.

[0114] Step 4: Sequential planar depth map integration;

[0115] After the training phase is completed, a method applicable to panoramic images is proposed to reconstruct the 3D layout through the integration of sequential planar depth maps.

[0116] First, the C2P conversion technology is used to generate the plane depth map of the ceiling, floor and each wall.

[0117] Subsequently, the walls are sorted from left to right and their respective plan depth maps are integrated in a sequential, step-by-step manner. This sequential integration ensures smooth transitions and precise connections between walls, laying the foundation for building an accurate 3D layout. During the integration process, the plan depth maps of the ceiling and floor are also taken into account to ensure that they are perfectly integrated with the wall depth maps.

[0118] Finally, through a delicate integration process, a complete and accurate 3D layout depth map is obtained. It is worth mentioning that this method does not rely on the assumption that walls are perpendicular to each other, so it can effectively handle complex indoor environments that deviate from the traditional Manhattan world assumption. This feature makes this method more flexible and applicable, and can cope with various complex indoor scenes.

[0119] The process of eliminating repeated predictions of wall categories based on the similarity of corresponding semantic labels is as follows Figure 2 As shown, (a) adjacent wall category predictions correspond to different walls. (b) adjacent wall category predictions are repeated, where only the category with higher activation value is retained.

[0120] The sequential planar depth map integration specifically includes:

[0121] Step 4.1: Determine the wall category in the image. Use the C2P conversion technology to generate the plane depth map of the ceiling, floor, and each wall, that is, use the plane category prediction C to define the threshold δ c , and select the predicted value greater than δ c However, this selection process may introduce duplication, as nearby predictions may potentially represent the same plane within the image. Figure 3 As shown in Figure 2, the predicted semantic labels of the adjacent plane categories are used to filter these duplicates. Whenever the wall category is selected, its two adjacent categories are checked. If any of these adjacent categories are also selected, the DICE loss is calculated to evaluate the similarity between the predicted semantic labels of these adjacent planes. The DICE loss exceeds a predefined threshold δ d indicates that adjacent predictions correspond to different walls, encouraging both to be retained. On the contrary, when the dice loss is lower than δ d When , it is suggested to keep only the wall category with higher prediction value. c =0.3,δ d =0.5.

[0122] Step 4.2. Sort the walls from left to right and integrate their respective planar depth maps in a sequential, step-by-step manner. Planar depth maps can be viewed as 3D surfaces in a coordinate system defined by pixel coordinates and a depth axis. For clarity, the integration of two planar depth maps within a 3D coordinate system is analyzed as the result of the intersection of two surfaces. The process begins with the depth map of the leftmost wall (note that the rightmost wall and the leftmost wall are parts of the same physical wall but are processed separately during the layout generation phase), and the depth maps of subsequent walls are gradually integrated. When integrating a new wall, an intersection line is formed between the new planar depth map and the existing planar depth map. In the actual room layout, the two walls should be located on both sides of this intersection line. The position of each wall relative to the intersection point is determined by referring to the centroid position derived from the plane semantic label. The area of ​​the planar depth map that extends beyond the intersection line is trimmed, resulting in an updated layout depth map. This process is applied repeatedly until the planar depth maps of all walls are integrated.

[0123] Step 4.3, through a fine integration process, a complete and accurate 3D layout depth map is obtained. Finally, the depth maps of the ceiling and floor are integrated into the layout depth map by using a pixel-by-pixel minimization operation. From the obtained layout depth map, a layout segmentation mask can be generated based on the plane category associated with each depth value.

[0124] To further enhance the robustness of the prediction, during the test phase, the input image is horizontally flipped and the alternative layout estimation result (horizontally flipped) is obtained in the same way. For the original image and the flipped image, the pixel-level matching rate between each layout segmentation mask obtained by sequential integration and S is evaluated, and the one with the higher matching rate is finally selected as the final layout estimation. This method ensures the reliability and accuracy of the prediction.

[0125] Example

[0126] Implementation of panoramic images. The panoramic image network is built on the basis of the proposed monocular depth estimation network, which uses SwinTransform as the encoder, the pyramid module to aggregate information, and the neural window FC-CRFS module as the decoder to predict the full depth map. The input panoramic image size is 256×512. Unlike the original model, the output resolution is adjusted to 128×256. The original decoder is copied twice to create two independent output branches for predicting the global plane visibility confidence map and the plane semantic label, respectively. During this copying process, the output channels of the plane semantic label prediction branch are adjusted to N+2, and the Sigmoid activation function is added to the output layer of each branch. The output branch for wall category prediction first uses a 1×1 convolution layer to reduce the channel dimension of the output features from the PPM head to 256. Then, bilinear interpolation is applied to adjust the resolution of the feature map to Then, 6 consecutive layers of 3×3 convolutions are applied with batch normalization and ReLU activation between each pair. In the experiments, N is set to 16 for cuboid layout datasets (such as Stanford2D-3D and PanoContext); N is set to 64 for general layout datasets (e.g., MatterportLayout and Zind).

[0127] A batch size of 4 was used and the AdamP optimizer was used with an initial learning rate of 0.0001. During layout training, the learning rate was decreased by a factor of 0.3 every 100 epochs. The training images were augmented with random horizontal flipping, panoramic horizontal rotation, and color jittering. The weighting coefficient was determined experimentally and was set to θ 1＝2 ,θ 2＝5 ,θ 3＝5 ,θ 4＝5 and θ 5＝2 .

[0128] Evaluation Metrics

[0129] Standard evaluation metrics for panoptic layout estimation are used, including: i) 3D intersection-over-union (3dioU), which is used to calculate the overlap between the predicted and true 3D room layouts; ii) 2D intersection-over-union (2dioU), which is used to calculate the overlap of floor shapes; iii) pixel error (PE), which is used to evaluate the pixel-level accuracy of the layout segmentation mask; iv) corner error (CE), which is used to measure the L2 distance between the predicted corners and the true corners; v) root mean square error (RMSE), which represents the average size of the errors in the layout depth map; and vi) threshold accuracy (δ1), which represents the percentage of pixel inliers with an error less than 1.25.

[0130] Results on the Panorama dataset

[0131] Four panoramic datasets are used to evaluate the performance of C2P-Net: Stanford2D-3D and PanoContext for cuboid layouts and MatterportLayout and Zind for general layouts. The following is an introduction to each dataset and experimental results.

[0132] Stanford2D-3D. This dataset is obtained by annotating the layout based on the Stanford2D3DS dataset. It contains 552 panoramas, all of which show a cube layout. The provided panorama depth map may contain some missing depth values. To address this issue, a colorization method is used to fill these missing depth values ​​before the training process. This process is also applied to the panoramic dataset MatterportLayout. The same training / validation / test split is followed. It is worth noting that some previous studies combined the training set of the Stanford 2D-3D dataset and the training set of the entire PanoContext dataset for training. However, due to the lack of complete depth maps in the PanoContext dataset, only the training set of the Stanford 2D-3D dataset is used for training.

[0133] Table I: Quantitative results of cuboid layout estimation based on panoramic images on the Stanford 2D-3D dataset:

[0134]

[0135] Table I shows the quantitative comparison results with the state-of-the-art methods. It can be seen that the proposed method (labeled as “C2P-Net”) outperforms all other methods even with less training data. In addition, the potential of incorporating external RGB-D data to enhance layout estimation is explored. A total of 1300 RGB-D panoramic images are collected from the original Stanford2D3DS dataset, excluding images in the Stanford2D3DS test set. Using this large dataset, the model is first pre-trained to learn the complete depth map and then fine-tuned on the training set of the Stanford2D3DS dataset. These results are labeled as “C2P-Net [with pre-training]”. These results are encouraging: the pre-trained model further improves the performance, demonstrating the effectiveness of the method in leveraging external RGB-D data to improve the complete depth map estimation, thereby improving the accuracy of layout estimation. The network prediction is able to fully exploit the rich external RGB-D dataset to enhance the network prediction performance, further improving the practicality and generalization ability of the network prediction.

[0136] Table II: Quantitative results of cuboid layout estimation based on panoramic images on the PanoContext dataset:

[0137]

[0138] PanoContext. This dataset is also a small-scale cuboid layout dataset containing 514 panoramas. No real-world full depth maps are provided in PanoContext, which poses a challenge for training on this dataset. To address this limitation, a full depth map prediction branch is adopted to learn the layout depth map while keeping the other components of the framework unchanged. The training / validation / test split of previous methods is also followed. During training, unlike previous studies that combined the entire stanford-2D3DS dataset, only the training split of PanoContext is used. The results in Table II show that the proposed method outperforms the existing methods on all evaluated metrics. This shows that even without access to the full depth map, the method is able to effectively learn from the layout depth map as a viable alternative.

[0139] Table III: Quantitative results of overall layout estimation based on panoramic images on the MatterportLayout dataset:

[0140]

[0141] MatterportLayout. This dataset contains 2,295 panoramic images carefully selected and annotated from the original Matterport3D dataset. It covers a wide range of room types, including room layouts with cuboid shapes and non-cuboid shapes (e.g., L-shaped, T-shaped rooms). The official train / validation / test split is adopted for evaluation. Quantitative results are shown in Table III. The proposed method achieves superior performance on all metrics. Compared with the previous state-of-the-art method, the proposed method significantly improves by 2.69% on 2dioU and 2.08% on 3dioU. In addition, the Matterport3D dataset is utilized, and 5,784 RGB-D panoramic images are collected from the buildings corresponding to the training set. Evaluation of the model pre-trained with these external RGB-D panoramic images shows that the performance is further improved.

[0142] The visualization results of panoramic layout estimation on the MatterportLayout dataset are shown in Figure 3 As shown, the intermediate outputs of the prediction are given, including a comprehensive depth map, a global plane visibility confidence, and a coarse layout segmentation derived from the predicted plane semantic labels. It is worth noting that in the MatterportLayout panoramic dataset, the learned plane visibility confidence tends to be concentrated in the boundary areas and occlusions between planes or between planes, while the plane visibility confidence values ​​are usually lower in the central area of ​​the plane. This trend may be attributed to the fact that the C2P transformation shows a higher sensitivity to depth errors in the central area of ​​the plane compared to the edge area. The layout estimation results obtained by this method are very close to the actual situation, even for more complex non-cuboid room layouts, as shown in columns 2-4 of the figure.

[0143] The qualitative comparison results of the MatterportLayout dataset are as follows Figure 4 As shown in the figure, the proposed C2P-Net is compared with LGT-Net and DOPNet. The boundaries of the room layout are shown on the left and the floor plan is shown on the right. The red line represents the true value and the green line represents the predicted value. The horizon depth is also shown below each panoramic image. The main differences are highlighted with dotted lines.

[0144] Figure 5 Qualitative comparisons with LGT-Net and DOPNet are given. The results in the first column show that the proposed method is not affected by obstacles on the right side of the panorama and always provides accurate predictions compared to LGT-Net and DOPNet. The results in the second and third columns show that the proposed method maintains its accuracy in complex scenes containing narrow walls in the panorama.

[0145] The qualitative comparison results of the Zind dataset are as follows Figure 5As shown in this paper, Zind, a large-scale panoramic dataset with layout annotations, provides a variety of room layout types, including Manhattan and non-Manhattan configurations. Experiments are performed using the "easy" subset, which contains samples that allow layout recovery from a single panoramic image. A formal training / validation / test split is adopted, where 24882 panoramas are used for training, 3080 for validation, and 3170 for testing. Since complete depth maps are not available in the Zind dataset, the same strategy implemented on the PanoContext dataset is used to overcome this limitation.

[0146] Table IV: Quantitative results of overall layout estimation based on panoramic images on the Zind dataset

[0147]

[0148] Table IV presents the quantitative comparison results on the Zind dataset. Our method achieves the best performance on all metrics, demonstrating its effectiveness in general room layout estimation. Figure 5 A qualitative comparison is shown. The proposed method consistently produces the most accurate results across all examples, demonstrating its effectiveness in handling complex environments and accurately detecting narrow walls.

[0149] Ablation

[0150] Qualitative comparison results of layout estimation and supervision plane visibility confidence, such as Figure 6 As shown, the comparison is evaluated on the MatterportLayout dataset. The red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines.

[0151] Table V: Quantitative comparison of panoptic layout estimation and planar visibility confidence supervision on the MatterportLayout dataset

[0152]

[0153] Plane visibility confidence is learned autonomously without direct supervision. To test the effectiveness of this learning strategy, pseudo-ground truth of plane visibility confidence is generated based on the pixel-wise similarity between the real comprehensive depth map and the layout depth map. As a comparison method, an L1 loss between the predicted global plane visibility confidence and the pseudo-ground truth is incorporated during training. A qualitative comparison of supervised and unsupervised plane visibility confidence on the MatterportLayout and MatterportLayout datasets is shown in Table V. A quantitative comparison of perspective layout estimation from learned layout depth maps and full depth maps is shown in Table VI. Figure 6Qualitative comparisons are shown. It can be seen that although the plane visibility confidence map learned through direct supervision covers most of the visible plane regions, the final layout estimation result is not as accurate as the map obtained through autonomous learning. This shows that forcing the network to fit all visible plane regions is unnecessary. Instead, letting the network independently identify and focus on the most credible visible plane regions leads to more favorable results.

[0154] The qualitative comparison results of the complete depth map and the layout depth map are as follows: Figure 7 As shown. The comparison is evaluated on the MatterportLayout dataset, the red line is the true layout, the green line is the prediction, and the main differences are highlighted with dotted lines.

[0155] Table VI: Quantitative comparison of panoptic layout estimation using learned layout depth maps and full depth maps on the MatterportLayout dataset.

[0156]

[0157] During the training process of C2P-Net, the depth information of the occluded part of the complete depth map is obtained. This helps the network to have a comprehensive understanding of the overall spatial structure of the indoor scene. In order to verify the impact of learning the depth information of the occluded part, a comparative experiment was conducted in which the branch originally used to predict the complete depth map was used to predict the layout depth map. Tables V and VI provide different quantitative comparison results on the MatterportLayout dataset. The results show that learning the complete depth map helps to obtain better layout estimates, proving that learning the depth information of the occluded part is beneficial for enhancing the network's understanding of the room layout. Figure 7 Qualitative comparison results are presented. The results show that the network trained to learn the full depth map is able to obtain plane visibility confidence maps that are able to distinguish planes from obstacles to a certain extent compared to learning the layout depth map. This suggests that learning the full depth map enables the network to automatically distinguish obstacles from planes, thus gaining a deeper understanding of the spatial structure of the scene. As a result, the final layout estimate is also more accurate.

[0158] Table VII: Quantitative comparison of panoptic layout estimation for different values ​​of N on the MatterportLayout dataset.

[0159]

[0160] In C2P-Net, wall categories (i.e., N) are used for panoramic images. The increase in the number of categories improves the network's ability to distinguish dense narrow planes, thereby pushing up the upper limit of the accuracy that the method can achieve. However, this expansion in categories also upgrades the challenge of accurate classification and expands the channels for learning plane semantic labels. As a result, some channels become prone to over-adaptation due to insufficient training. In Table VII, the overall layout estimation results on the panoramic dataset MatterportLayout are compared. The results show that when N = 64, the layout estimation of all indicators is optimal.

[0161] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A panoramic image C2P conversion and layout generation method for room layout estimation, comprising: Step 1, parameter representation of plane depth in equirectangular panoramic images; Storing the distance from the camera center position to each point in each direction, the pixel p = (u, v) with depth value R in the equirectangular panoramic image is converted to a 3D point in the spherical coordinate system centered on the camera Step 2: Based on the plane depth parameterization, C2P conversion is introduced; Step 3: Reconstruct the layout of each plane in the panoramic image by predicting its depth map. According to the C2P transformation, a complete depth map is predicted, the main planes in the image are identified, and the visibility confidence of each plane is determined. Step 4: Sequential planar depth map integration.

2. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: In step 1 Where r is point P S is the radial distance to the camera center, and and θ are determined to be expressed as Where w and h are the width and height of the panoramic image in pixels, respectively. The horizontal coordinate u and vertical coordinate v of the pixel are the column and row numbers in the image pixel coordinates, respectively. Then P S Further conversion to Cartesian coordinates yields P c : Assuming the point lies on a plane in 3D space, the equation is is the normal vector, is the distance to the camera center; By combining formula (2), the relationship between the plane depth and pixel coordinates in the panoramic image can be derived as follows: in Define P = [φψχ] T is the depth parameter of the plane in the panoramic image, which can be used to determine the depth map of any plane. Formula (3) and (1) together establish the mathematical relationship between the pixel coordinates and depth value of the plane in the panoramic image, thereby allowing the plane depth to be reconstructed in the panoramic image using the plane depth parameter.

3. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The C2P transformation introduced in step 2 includes: taking a plane denoted as k as an example, given a complete depth map denoted as R and a depth map denoted as V k The corresponding visibility confidence map of plane k is obtained by C2P transformation, and the goal of C2P transformation is to derive P k The plane depth parameters of plane k are used to facilitate plane depth reconstruction, using pixel coordinates derived from R and their associated inverse depth values, which are the inverse of the depth, to form two matrices, X∈R n ×3 and Y∈R n .

4. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 3 further includes: decomposing the complex visibility confidence prediction task into two subtasks, including: Step 3.1: Predict the semantic label of each plane to identify the main planes in the scene; Step 3.2: Repeat the last output branch to predict a global plane visibility confidence map, which intuitively reflects the visibility of all main planes.

5. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 4, characterized in that: The step 3.1 further includes a four-task learning framework for accurate 3D layout estimation; the C2P network architecture of the panoramic image network prediction consists of a feature encoder and four independent output branches, and the four independent output branches each complete four different prediction tasks, and the four different prediction tasks include generating a complete depth map, identifying the wall type in the image, assigning a semantic label to each plane, and generating a confidence map of global plane visibility; specifically including: i) Generate a complete depth map. For a given indoor panorama, use the first output branch to predict the complete depth map; ii) Identify the wall type in the image. A panoramic image of a room usually includes the ceiling, floor, and multiple walls. However, when labeling and identifying these elements, the definition of the ceiling and floor is relatively clear, while the definition of the wall is relatively vague and ambiguous. iii) Assign semantic labels to each plane. For a given indoor panorama, the predicted plane semantic label is named S∈R h×w×(N+2) , N is the number of anchor points uniformly distributed along the horizontal direction of the panoramic image, and 2 represents the ceiling and the ground; iiii) generating a global plane visibility confidence map, the panoramic image network can reconstruct a plane depth map according to the plane visibility confidence map, thereby allowing the panoramic image network to autonomously determine the visibility confidence according to actual needs.

6. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 5, characterized in that: The C2P network architecture in the panoramic image network prediction is based on a monocular depth estimation network. The C2P network architecture of the panoramic image uses SwinTransform as an encoder, a pyramid module to aggregate information, and a neural window FC-CRFS module as a decoder to predict a complete depth map.

7. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 1, characterized in that: The step 4 further comprises: Step 4.1, determine the wall category in the image, and use the C2P conversion technology to generate the plane depth map of the ceiling, floor and each wall; Step 4.2, sort the walls from left to right and integrate their respective plan depth maps in a sequential and step-by-step manner; Step 4.3: Through a sophisticated integration process, a complete and accurate three-dimensional layout depth map is obtained, and the depth maps of the ceiling and floor are integrated into the layout depth map by using a pixel-by-pixel minimization operation.

8. The panoramic image C2P conversion and layout generation method for room layout estimation according to claim 7, characterized in that: The step 4.2 further includes: analyzing the integration of two planar depth maps in the three-dimensional coordinate system, starting the process with the depth map of the leftmost wall as a result of the intersection of two surfaces, and gradually integrating the depth maps of subsequent walls; determining the position of each wall relative to the intersection point with reference to the centroid position derived from the plane semantic label; trimming the area of ​​the planar depth map that extends beyond the intersection line, thereby generating an updated layout depth map; this process is repeatedly applied until the planar depth maps of all walls are integrated.

Images