Multi-view panoramic image indoor layout prediction method fusing semantics and planarity

Abstract

The invention provides a multi-view panoramic image indoor layout prediction method fusing semantics and planarity. The method comprises the following steps: step 1, carrying out three-dimensional semantic segmentation combined with three-dimensional point cloud reconstruction and a multi-view geometry and two-dimensional segmentation large model; step 2, obtaining a pure structured point cloud; step 3, multi-plane detection and classification; 4, extracting an initial contour; 5, generating a wall line based on plane fitting; step 6, performing multi-label graph segmentation guided by the two-dimensional contour and the wall surface; step 7, extracting a non-Manhattan ceiling line; and step 8, stretching the two-dimensional wall line along the vertical direction on the basis of the contour of the dimensional plane graph in combination with the vertical height of the room, generating an intersection line of the wall surface and the ceiling and an intersection line of the wall surface and the ground, and finally outputting a complete indoor three-dimensional vectorization layout model. According to the method, the layout integrity of the complex scene is improved, the boundary recovery precision of the occlusion area is improved compared with that of a traditional projection method, and the detection accuracy is improved by matching with multi-model RANSAC plane detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, specifically to a method for predicting indoor layouts from multi-view panoramic images that integrates semantics and planarity. Background Technology

[0002] Interior layout estimation is a fundamental problem in computer vision and graphics, aiming to recover the geometry of a room's walls, floors, and ceilings from images or point clouds. It has wide applications in augmented reality, interior navigation, and interior design.

[0003] Existing technologies typically rely on single panoramic images for deep learning predictions. However, single-view images are easily obstructed by indoor furniture and decorations, leading to incomplete layout predictions or geometric distortions. While multi-view stereo vision can reconstruct 3D point clouds, the directly generated point clouds often contain a large amount of noise (such as a mixture of furniture and wall point clouds), and due to the lack of indoor textures, the reconstructed walls are often uneven, even showing holes. Furthermore, traditional point cloud projection-based methods, when dealing with complex scenes, often directly project all point clouds onto the ground, ignoring the interference from ground debris (such as carpets and table legs), resulting in inaccurate wall outlines. Moreover, simple outline extraction cannot guarantee the flatness and verticality of the walls, making it difficult to generate high-quality vectorized layouts that conform to the assumptions of Manhattan or non-Manhattan worlds.

[0004] Multi-view panoramic technology, with its wide field of view, has gradually replaced single-view methods, becoming the mainstream research direction for solving complex occlusion and improving scene integrity. However, layout prediction based on multi-view panoramic technology still faces multiple technical challenges. First, the depth of geometric and semantic fusion is insufficient. Existing mainstream methods (such as multi-label graph cut) rely heavily on the initial projection of deep learning models, failing to fully utilize the rich semantics and precise planar structure in point clouds. This leads to blurred boundaries and structural distortion, especially in non-Manhattan structures or severely occluded areas. Second, data quality also affects the accuracy of reconstruction. MVS-reconstructed point clouds generally suffer from problems such as "flying point" noise and missing textures, directly interfering with the accuracy of subsequent planar detection. Finally, there is a lack of a complete optimization framework. Although datasets such as HM3D provide rich samples, there is still no mature solution for how to extract semantic priors (such as walls, doors, and windows) from noisy point clouds and deeply embed planarity constraints into the energy function of layout inference.

[0005] Therefore, there is a need for an interior layout prediction method that can effectively utilize semantic information to filter out interference, leverage ceiling integrity to complement missing floor elements, and optimize wall connection relationships through geometric constraints. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multi-view panoramic image indoor layout prediction method that integrates semantics and planarity, addressing the problems of severe occlusion, high point cloud noise, and non-straight wall line extraction in existing indoor layout reconstruction.

[0007] The method includes the following steps: Step 1: Perform 3D semantic segmentation by combining 3D point cloud reconstruction with multi-view geometry and a large 2D segmentation model; Step 2: Combine semantic and prior information to filter out flying points and obtain a pure structured point cloud; Step 3, Multi-plane detection and classification; Plane detection is performed on the filtered point cloud. The multi-model random sampling consensus (RANSAC) algorithm is used, combined with semantic labels and normal vector constraints, to extract the wall plane, ceiling plane and ground plane respectively. Step 4, extract the initial contour: calculate the average height of the ground plane and the room height, project the detected ceiling point cloud vertically onto the ground plane, and fuse the projected ceiling point cloud, the original ground point cloud and the wall slice point cloud into an enhanced ground projection map; use morphological closing operation to connect the broken wall contours, fill the projection holes, extract the approximate geometric boundary of the room through the contour detection algorithm, and generate the initial layout contour. Step 5: Wall line generation based on plane fitting; For each detected wall point cloud cluster, perform three-dimensional plane fitting to obtain the wall's normal vector and center point. On the two-dimensional projection plane, use principal component analysis or line fitting to calculate the straight line equation of each wall. Calculate the intersection points of the straight line equations of adjacent or intersecting walls as candidate points for wall corners, and generate wall line segments with topological relationships. Step 6, 2D contour and wall-guided multi-label cut: Construct an energy function, use high-precision wall lines and initial contours, and generate a geometrically consistent ground and wall intersection contour by minimizing the energy function; Step 7, Non-Manhattan Ceiling Line Extraction: Introduce the geometric assumption that the wall is vertical and intersects the ceiling. Based on the vertical height of the room, calculate and generate the corresponding ceiling outline. Step 8: 3D vectorized layout generation; Based on the 3D planar outline and combined with the room's vertical height, the 2D wall lines are stretched along the vertical direction to generate the intersection lines of the wall and ceiling, as well as the intersection lines of the wall and floor, ultimately outputting a complete 3D vectorized interior layout model.

[0008] Step 1 includes:

[0009] First, multi-view panoramic images from the HM3D dataset are loaded. Taking into account the isometric cylindrical projection characteristics of panoramic images, a spherical camera model is configured in the open-source 3D reconstruction software Colmap to adapt to the panoramic field of view. Then, feature extraction and sparse reconstruction are performed to generate an initial 3D point cloud reflecting the indoor geometry. Then, the 2D segmentation large model SAM3 (SegmentAnything Model 3) is used to extract multi-view image features. Combined with multi-view geometric mapping, semantic labels for each 3D point are obtained, effectively identifying doors, windows, walls, ceilings, floors and indoor furniture objects, providing a semantic basis for subsequent processing.

[0010] Step 2 includes:

[0011] Perform bidirectional filtering based on semantic tags to identify the point cloud corresponding to the door and window areas and remove noise points behind the doors and windows;

[0012] Then, a statistical filtering algorithm is applied to the filtered structural point cloud. Let the neighborhood radius of the point cloud be r. The number of neighboring points within radius r for each point is counted, and isolated points with fewer than a threshold k are removed. The formula is as follows:

[0013] (1),

[0014] in, To represent any point in the input point cloud, Given the input point cloud set, For point In radius The neighborhood point set within, This is a function for counting the number of points in the neighborhood. The threshold for the number of neighboring points;

[0015] The final output is a clean, structured point cloud containing walls, ceilings, floors, and other elements such as windows and door frames. .

[0016] Step 3 includes:

[0017] First, use the semantic segmentation results to process the preprocessed point cloud. Divided into wall surfaces ,ground and ceiling Three subsets are used, and then an adaptive parameter adjustment mechanism is introduced for point clouds of different densities, by constructing... Tree calculation of local point cloud average spacing Dynamically set the RANSAC distance threshold With cluster radius Before RANSAC fitting, a density-based spatial clustering algorithm, DBSCAN, is introduced to project the wall point cloud onto... Planar clustering separates disconnected walls into independent clusters. Iterative RANSAC detection is performed within each cluster, where Indicates the k-th cluster;

[0018] For any subset of point clouds, the planar model is defined by the normal vector. and distance parameters It is determined that the equation is satisfied:

[0019] (2),

[0020] Where T represents the transpose; a, b, and c represent the three components of the plane normal vector n; and x, y, and z represent the coordinates of any point in space.

[0021] A hypothetical model is generated by randomly sampling three interior points, and the points are calculated. Euclidean distance to the plane :

[0022] (3),

[0023] if Then determine Let it be an interior point;

[0024] Maximize the number of interior points through iteration To obtain the optimal planar model, points within the planar model are removed from the original point cloud. The remaining point cloud is then tested until the number of remaining points is insufficient or the preset maximum number of planar models is reached. During this process, geometric constraints are introduced to filter invalid planar models: for walls, the normal vector must be perpendicular to the surface. And the height span meets the requirements For the ground and ceiling, the normal vector is required to be approximately parallel to... axis; This is the verticality threshold, used to determine whether a plane is a wall. Since a wall should be vertical, its normal vector should be close to horizontal (i.e., the z-component is very small). This threshold is used to filter out planes that are not vertical enough. It is the minimum height threshold. This is used to ensure that the wall surface being detected has sufficient height, thereby filtering out small-sized flat surfaces such as tabletops and low cabinets.

[0025] Extract the set of all detected wall normal vectors Select the normal vector of the main wall surface with the most interior points. As a reference benchmark, for other wall surfaces Calculate the angle between wall surface j and the principal direction. :

[0026] (4),

[0027] in Let m be the normal vector of the m-th wall. near Then, the Gram-Schmidt orthogonalization method is used to enforce constraints. orthogonal to .

[0028] Step 4 includes: using the ceiling point cloud and ground point cloud detected by the RANSAC algorithm in step 3, and analyzing the Y coordinate of the point cloud to determine the highest point of the room. and lowest point The vertical distance between them is used to obtain the overall height of the room. :

[0029] (5),

[0030] Define distance threshold :

[0031] (6),

[0032] Based on distance threshold The point cloud of the entire room is divided into the following three parts:

[0033] Ceiling: Points in the region near the minimum height at the Y-coordinate:

[0034] (7),

[0035] Floor: Points in the region near the maximum height at the Y-coordinate:

[0036] (8),

[0037] Other: Points located in the middle area;

[0038] Obtain a more accurate ground reference:

[0039] (9),

[0040] in This represents the estimated ground plane height. It retrieves the coordinates of 90% of the Y-coordinates of the ground point cloud set. It is the set of Y coordinates of the ground point cloud set;

[0041] The average height of the ceiling is estimated using the median of the Y-coordinates of the ceiling point cloud:

[0042] (10)

[0043] in This represents the estimated ceiling height. This is to obtain the coordinates of the median position of the ceiling point cloud set. It is the set of Y coordinates of the ceiling point cloud set;

[0044] Using orthogonal projection, the ceiling point cloud is projected onto the ground plane: First, the Y coordinates of all copied ceiling points are uniformly set to the precise floor plane height calculated in the previous step. For any ceiling point Ceiling point New coordinates after projection onto the floor plane for:

[0045] (11),

[0046] in , These are the horizontal coordinates of the original point. It is the Y-coordinate of the original point;

[0047] The 3D projected point cloud is then converted into a binary image from a bird's-eye view. Morphological optimization is performed on the binary image using a rectangular structuring element to perform morphological closing operations. First, dilation is performed to connect the broken wall contours, and then erosion is performed to fill the voids. The formula is as follows:

[0048] (12)

[0049] in It is the input boundary image. It is a structural element. Indicates expansion. Indicates corrosion;

[0050] The contour retrieval mode and contour approximation method are used to extract the room boundary contours in the image, and small noise contours are removed by area filtering; and polygon fitting is performed on the extracted contours to transform jagged pixel edges into straight walls.

[0051] Finally, the pixel coordinates on the image plane are... Convert to real-world 3D coordinates It outputs the initial layout outline, and the transformation logic is as follows:

[0052] X-axis coordinate for:

[0053] (13)

[0054] Z-axis coordinate for:

[0055] (14)

[0056] Y-axis coordinate for:

[0057] (15)

[0058] in, , These are the column coordinates and row coordinates of the image, respectively. It's resolution. It is the pixel value of the white space at the edge of the image. It is the minimum coordinate offset of the original point cloud in the X-axis direction. It is the minimum coordinate offset of the original point cloud in the Z-axis direction. It's the floor height.

[0059] Step 5 includes: applying cloud-like dots to each wall surface. The wall point cloud is sliced ​​to extract points in the middle area of ​​the wall height. The slice range is:

[0060] (16)

[0061] in It is a point cloud after slicing. It is the minimum height of the slice. is the highest height of the slice, and h is the height of the wall.

[0062] Projecting the point cloud of the wall slices onto the XZ plane yields a two-dimensional point set. Then, PCA principal direction analysis is performed to obtain the covariance. :

[0063] (17)

[0064] Where N represents the total number of points projected onto the XZ plane from the wall surface. It is the coordinate vector of the i-th point in the point set;

[0065] right Perform eigenvalue decomposition The eigenvector corresponding to the largest eigenvalue The principal direction of the wall, and the vector perpendicular to the wall. It is a two-dimensional normal vector, then based on the center of the point set. Establish the two-dimensional straight line equation of the wall:

[0066] (18)

[0067] in , , , , These are the components of the two-dimensional normal vector. It is a constant term calculated through the center point;

[0068] For any two adjacent wall planes, a straight line and Solve the system of equations to find the intersection of the projected lines, which can be used as candidate corner points. The formula is as follows:

[0069] (19)

[0070] If the determinant If there is an intersection, then the intersection point exists. Calculate the intersection point. to the original wall line endpoint Euclidean distance ;like Determine the location of the two walls at point Connecting forces the corresponding endpoints to the intersection. At the same time, update the wall line endpoints to This automatically closes the room outline; Det is an abbreviation for Determinant, representing determinant, and snap is the preset distance judgment threshold.

[0071] Step 6 includes: setting the initial two-dimensional contour point set As nodes in the graph, the set of wall lines optimized by PCA fitting is selected. As a set of candidate labels; for each contour point Assign an optimal wall label ;in This represents the nth contour point; This represents the straight line on the k-th wall surface;

[0072] Constructing the energy function :

[0073] (20)

[0074] in, It is the set of all adjacent pairs of points. They are two adjacent points on the initial two-dimensional contour. It is the specific wall label assigned to point p. It is the specific wall label assigned to point q. These are the planar coordinates of point p; data item Used to measure contour points Straight label to the assigned wall Geometric distance; label The equation of the straight line is Defined as:

[0075] (twenty one),

[0076] Smoothing Term Measuring neighboring nodes Consistency in label assignment;

[0077] Using the Potts model, the frequent changes in the penalty label are penalized:

[0078] (twenty two),

[0079] Where σ is the smoothing penalty constant;

[0080] Finally adopted The algorithm iteratively minimizes the energy function and, based on the optimized graph cut result, identifies elements belonging to the same label. The continuous set of points is fitted to form the final floor line.

[0081] Step 7 includes: first, performing ceiling plane fitting, using the ceiling point cloud clusters detected in step 3. The RANSAC algorithm is used to fit the optimal ceiling plane equation. Let the plane equation be... ,in It is the normal vector. The distance parameter is obtained by iteratively solving for the distance parameter by maximizing the number of interior points. If the fitting fails or the point cloud is too small (too small if the point cloud is less than 500), the solution is to use the global room height calculated in step 4. and ground height Calculate a fixed ceiling height ;

[0082] Then use each ground contour point output in step 6 Find the intersection point of the ground contour point and the ceiling plane in the vertical direction. If a planar equation model is used, the corresponding ceiling height is... The calculation formula is:

[0083] (twenty three),

[0084] The calculated height coordinates are combined with the planar coordinates to obtain the corresponding three-dimensional ceiling contour point set. This forms a ceiling line that is topologically consistent with the ground contour but highly accurate in matching the point cloud distribution; It is the X-axis coordinate of the i-th ground contour point in the world coordinate system. is the Z-axis coordinate of the i-th ground contour point in the world coordinate system, and N represents the total number of discrete points that constitute the entire room's ground contour.

[0085] Step 8 includes:

[0086] By traversing the sequence of ground contour points, the cosine of the angle between adjacent line segments is calculated. For three consecutive points... Construct vectors sum vector Calculate the dot product of the normalized vectors:

[0087] (twenty four),

[0088] If the included angle Less than the preset threshold Then the three points are determined to be collinear. Redundant points are removed; otherwise, they are retained. As key geometric corner points of the house; repeat until the outline converges, resulting in a simplified set of corner points. ; It is the two-dimensional coordinate point of the corner point;

[0089] Then, based on the ceiling plan model or height information from step 7, determine the corresponding ground vertex. and ceiling apex ;connect and Generate vertical line segments;

[0090] Finally, based on the topological connection order of the corner points, adjacent ground vertices are connected in sequence to form a closed ground outline, and adjacent ceiling vertices are connected to form a closed ceiling outline, thus forming a complete three-dimensional vectorized layout model of the interior.

[0091] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0092] This invention achieves high-precision, occlusion-resistant indoor 3D layout reconstruction by introducing semantic segmentation to filter interference points, using ceiling point cloud projection to compensate for the ground structure, directly generating high-quality wall lines through wall plane fitting and intersection, and combining multi-label graph cut for global regularization.

[0093] Beneficial effects: While retaining the advantages of global information from multi-view panoramic images, this invention effectively solves four major challenges in indoor layout prediction by employing a technical approach that deeply integrates semantic filtering and geometric constraints.

[0094] Structural completion of occluded areas: By compensating for the vertical projection of the unobstructed point cloud on the ceiling and combining it with the fusion of the point cloud slices on the wall, the structural loss caused by the occlusion of furniture and decorations on the ground is filled, which improves the integrity of the complex scene layout and improves the accuracy of boundary restoration of occluded areas compared with traditional projection methods.

[0095] Precise noise interference suppression: By eliminating unstructured object point clouds through SAM3 semantic segmentation, noise behind doors and windows and point cloud discretization errors are filtered out, thus improving the purity of structured point clouds. Combined with multi-model RANSAC plane detection, the detection accuracy is greatly improved.

[0096] Precise wall topology construction: Based on wall plane fitting and intersection calculation, the upper and lower contours and intersecting edges of the wall are directly generated, solving the problems of non-straight wall lines and misaligned corners in traditional projection. This improves the straightness of wall lines and the accuracy of topological connections, and reduces the corner alignment error in non-Manhattan scenes.

[0097] Global optimization ensures reliability: Global regularization of multi-labeled graph cuts corrects local geometric deviations, making the layout conform to the building structure logic and improving the overall geometric accuracy. Attached Figure Description

[0098] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0099] Figure 2 This is a flowchart of the three-dimensional semantic segmentation process that combines multi-view geometry with a large two-dimensional segmentation model.

[0100] Figure 3 This is a schematic diagram of the original point cloud.

[0101] Figure 4 To segment large model instances using two-dimensional segmentation.

[0102] Figure 5 This is a schematic diagram of the filtered, clean point cloud (blue represents the ceiling, red represents the floor, and green represents the walls).

[0103] Figure 6 To generate an initial contour diagram.

[0104] Figure 7 This is a schematic diagram of the optimized floor line.

[0105] Figure 8 This is a result image based on the multi-view panoramic and multi-label cut method.

[0106] Figure 9 This is a graph showing the results of this method. Detailed Implementation

[0107] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0108] like Figure 1 As shown, this embodiment of the invention provides a method for predicting indoor layouts using multi-view panoramic images that integrates semantics and planarity, including the following steps:

[0109] Step 1, as follows Figure 2 As shown, 3D semantic segmentation is achieved by combining 3D point cloud reconstruction with multi-view geometry and a large 2D segmentation model; a sequence of indoor multi-view panoramic images is acquired, and an initial 3D dense point cloud is generated using multi-view geometry technology; the point cloud is segmented into instances using the semantic segmentation model (Segment Anything Model 3, SAM3) to identify doors, windows, walls, ceilings, floors, and indoor furniture objects.

[0110] Step 2: Combine semantic and prior information for fly point filtering; based on semantic labels, remove noisy point clouds behind doors and windows as well as point clouds of unstructured objects (furniture, clutter). Combine with prior information about the interior structure, perform statistical filtering to denoise the remaining structured point clouds, filtering out fly points and discrete noise to obtain a clean structured point cloud.

[0111] Step 3, Multi-plane detection and classification: Perform plane detection on the filtered point cloud, using the multi-model RANSAC algorithm, combined with semantic labels and normal vector constraints, to extract multiple wall planes, ceiling planes and ground planes respectively.

[0112] Step 4, Initial Contour Extraction: Calculate the average height of the ground plane and the room height, and project the detected ceiling point cloud vertically onto the ground plane. Merge the projected ceiling point cloud, the original ground point cloud, and the wall slice point cloud into an enhanced ground projection map. Use morphological closing operations to connect broken wall contours, fill projection gaps, and extract the approximate geometric boundaries of the room using a contour detection algorithm to generate the initial layout contour.

[0113] Step 5: Wall line generation based on planar fitting; For each wall point cloud cluster detected in Step 3, perform 3D planar fitting to obtain the wall's normal vector and center point. On the 2D projection plane, calculate the linear equation of each wall using principal component analysis or linear fitting. Calculate the intersection points of the linear equations of adjacent or intersecting walls, using these as candidate points for wall corners to generate wall segments with topological relationships.

[0114] Step 6: Perform multi-label cuts on the 2D contour and wall-guided graph; construct an energy function, and use the high-precision wall line generated in Step 5 and the initial contour extracted in Step 4 to generate a geometrically consistent ground and wall intersection contour by minimizing the energy function.

[0115] Step 7, Non-Manhattan Ceiling Line Extraction: Based on the ground contour obtained in Step 6, the geometric assumption that the walls are vertical and intersect with the ceiling is introduced. According to the room vertical height calculated in Step 4, the corresponding ceiling contour line is calculated and generated.

[0116] Step 8: 3D vectorized layout generation; Based on the 2D plan outline obtained in Steps 6 and 7, and combined with the room vertical height calculated in Step 4, the 2D wall lines are stretched along the vertical direction to generate the intersection lines of the wall and ceiling as well as the intersection lines of the wall and floor, and finally output a complete 3D vectorized interior layout model.

[0117] The specific process of 3D semantic segmentation combining 3D point cloud reconstruction with multi-view geometry and large model described in step 1 is as follows:

[0118] First, load multi-view panoramic images from the HM3D dataset. Taking into account the equidistant cylindrical projection characteristics of panoramic images, configure a spherical camera model in Colmap to adapt to the panoramic field of view. Then, perform feature extraction and sparse reconstruction to generate an initial 3D point cloud reflecting the indoor geometry, such as... Figure 3 As shown in the image. Then, the SAM3 large-scale 2D segmentation model is used to extract multi-view image features, and combined with multi-view geometric mapping to obtain semantic labels for each 3D point, effectively identifying objects such as doors, windows, walls, ceilings, floors, and indoor furniture, providing a semantic foundation for subsequent processing. For example... Figure 4 The image shows a two-dimensional segmentation plot of a large model instance, where id represents the ID and prob represents the confidence level.

[0119] The specific process of flitting semantics and prior information for filtering out flying points in step 2 is as follows:

[0120] Bidirectional filtering is performed based on semantic tags to identify the point clouds corresponding to door and window areas, removing noise points behind doors and windows (i.e., non-structural points in the transparent areas of doors and windows) to avoid the impact of door and window openings on the integrity of the wall. Then, a statistical filtering algorithm is applied to the filtered structural point cloud. Let the neighborhood radius of the point cloud be r, count the number of neighborhood points of each point within the radius r, and remove isolated points with a neighborhood number less than a threshold k. The formula is as follows:

[0121] (1),

[0122] in, For the filtered structural point cloud, For point In radius The neighborhood point set within, This is a function for counting the number of points in the neighborhood. The threshold for the number of neighboring points (in this invention, k=10, r=0.05m).

[0123] The final output is a clean, structured point cloud containing walls, ceiling, floor, and other data. ,like Figure 5 As shown.

[0124] The specific process of the multi-model RANSAC plane detection algorithm described in step 3 is as follows:

[0125] First, use the semantic segmentation results to process the preprocessed point cloud. Divided into wall surfaces ,ground and ceiling Three subsets are used, and then for point clouds of different densities, the algorithm introduces an adaptive parameter adjustment mechanism by constructing... Tree calculation of local point cloud average spacing Dynamically set the RANSAC distance threshold With cluster radius To balance detection accuracy and robustness, and considering the numerous and complex wall structures, the DBSCAN density clustering algorithm is introduced before RANSAC fitting. The wall point cloud is then projected onto... Planar clustering separates disconnected walls into independent clusters. Iterative RANSAC detection is performed within each cluster.

[0126] For any subset of point clouds, assume the planar model is composed of normal vectors. and distance parameters It is determined that the equation is satisfied:

[0127] (2),

[0128] The algorithm generates a hypothetical model by randomly sampling three interior points and calculates the points. Euclidean distance to the plane :

[0129] (3),

[0130] if Then determine Let these be interior points. The algorithm maximizes the number of interior points through iteration. To obtain the optimal planar model, points within this plane are removed from the original point cloud. The remaining point cloud is then tested again until the number of remaining points is insufficient or the preset maximum number of planes is reached. During this process, geometric constraints are introduced to filter invalid planes: for walls, the normal vector must be perpendicular to the surface. And the height span meets the requirements For the ground and ceiling, the normal vector is required to be approximately parallel to... axis.

[0131] To address the plane tilting issue caused by noise, global orthogonalization was introduced. This involved extracting the set of all detected wall normal vectors. Select the normal vector of the main wall surface with the most interior points. As a reference benchmark. For other wall surfaces Calculate the angle between it and the principal direction. :

[0132] (4),

[0133] like near Then, the Gram-Schmidt orthogonalization method is used to enforce constraints. orthogonal to This helps to repair structural distortions caused by noise.

[0134] Finally, the detected point clouds of the ceiling, floor, and walls are classified and saved to prepare for the initial outline and wall projection later.

[0135] The specific process for generating the initial contour in step 4 is as follows:

[0136] Taking advantage of the unobstructed nature of the ceiling, the ceiling structure is mapped to the ground via vertical projection, compensating for structural gaps caused by ground occlusion. Using the ceiling and ground point clouds detected by the RANSAC algorithm in step 3, the highest point of the room is determined by analyzing the Y-coordinate of the point clouds. (Approximate ground height) and lowest point The overall height of the room can be calculated by the vertical distance between (approximately the ceiling height) and (approximately) the ceiling height. The formula is:

[0137] (5),

[0138] To separate the ceiling and floor from the entire point cloud, a distance threshold (0.15 in this invention) is defined as a percentage of the total room height. Formula:

[0139] (6),

[0140] Based on this threshold, the entire room's point cloud is divided into three parts:

[0141] (1) Ceiling: A point in the region where the Y-coordinate is near the minimum height;

[0142] (7),

[0143] (2) Floor: A point in the region near the maximum height with Y coordinate.

[0144] (8),

[0145] (3) Others: Points located in the middle area, usually walls, furniture, etc.

[0146] Meanwhile, to obtain a robust floor plane height, 90% of the floor point cloud Y coordinate was calculated, effectively eliminating a small number of abnormal high point clouds caused by ground debris or skirting board noise, thus obtaining a more accurate ground reference.

[0147] (9),

[0148] Similarly, using the median of the Y-coordinates of the ceiling point cloud to estimate the average ceiling height can resist the interference of outliers:

[0149] (10)

[0150] Then, an orthogonal projection method was used to project the ceiling point cloud onto the ground plane; first, the Y coordinates of all copied ceiling points were uniformly set to the precise floor plane height calculated in the previous step. For any ceiling point The new coordinates after its projection onto the floor plane for:

[0151] (11),

[0152] The 3D projected point cloud (with all points on the same Y-plane) is then converted into a binary image from a bird's-eye view. Since the point cloud projection may contain noise or scanning blind spots, directly extracting the contours will produce many fragments. Morphological optimization of the binary image is necessary. A rectangular structuring element (7×7 kernel) is used to perform morphological closing operations. First, dilation is performed to connect the broken wall contours, and then erosion is performed to fill the voids. The formula is as follows:

[0153] (12)

[0154] in Indicates expansion. It indicates corrosion.

[0155] The contour retrieval mode and contour approximation method are used to extract the room boundary contours in the image, and small noise contours are removed by area filtering; then polygon fitting is performed on the extracted contours to transform jagged pixel edges into straight walls.

[0156] Finally, the pixel coordinates on the image plane are... Convert to real-world 3D coordinates It outputs the initial layout outline, and the transformation logic is as follows:

[0157] X-axis coordinate:

[0158] (13)

[0159] Z-axis coordinate (corresponding to the vertical direction v of the image):

[0160] (14)

[0161] Y-axis coordinate (height):

[0162] (15)

[0163] in: These are image pixel coordinates. It refers to the resolution (meters per pixel). It is the pixel value of the white space at the edge of the image. It is the minimum coordinate offset of the original point cloud. It's the floor height.

[0164] The final morphological operation successfully connected the wall fractures caused by the scanning blind spots and filled the projection holes, generating a continuous initial two-dimensional outline map reflecting the overall geometry of the room, as shown below. Figure 6 As shown.

[0165] The specific process of generating wall lines based on planar fitting wall projection in step 5 is as follows:

[0166] Dot clouds on each wall To avoid interference from baseboards and ceilings, the wall point cloud is sliced, extracting points in the middle area of ​​the wall height (avoiding noise at the top and bottom of the wall). The slice range is:

[0167] (16)

[0168] Projecting the point cloud of the wall slices onto the XZ plane yields a two-dimensional point set. Then, PCA principal direction analysis is performed on it to obtain the covariance. :

[0169] (17)

[0170] right Perform eigenvalue decomposition The eigenvector corresponding to the largest eigenvalue The main direction of the wall. The vector perpendicular to it. It is a two-dimensional normal vector. Then, based on the center of the point set... This allows us to establish the two-dimensional straight line equation of the wall:

[0171] (18)

[0172] in , , .

[0173] Because each wall is fitted independently, the endpoints of adjacent walls typically do not close perfectly, but rather have gaps or intersections. Therefore, for any two adjacent wall planes, the straight lines... and Solve the system of equations to find the intersection of the projected lines, and use the intersection as a candidate corner point. The formula is as follows:

[0174] (19)

[0175] If the determinant If the intersection points are equal, then an intersection point exists. Calculate the intersection point. to the original wall line endpoint Euclidean distance .like If the two walls are considered to be connected at this point, the corresponding endpoints are forcibly moved to the intersection. At the same time, update the wall line endpoints to This automatically closes the room's outline.

[0176] The specific process of multi-label cutting of the two-dimensional contour and wall guidance described in step 6 is as follows:

[0177] The initial two-dimensional contour point set extracted in step 4 These points, serving as nodes in the graph, roughly describe the shape of the room, but due to noise, they exhibit jitter or non-linearity. Simultaneously, the set of wall lines optimized by PCA fitting in step 5 is selected. As a set of candidate labels. For each contour point. Assign an optimal wall label This ensures that the point is assigned to the correct wall surface.

[0178] Constructing the energy function It includes data items and smoothing items:

[0179] (20)

[0180] Among them: data items Used to measure contour points To the wall straight label to which it was assigned The geometric distance. The smaller the distance, the lower the cost. Tags The equation of the straight line is (Calculated from step 6), defined as:

[0181] (twenty one),

[0182] Smoothing Term Measuring neighboring nodes Consistency in label assignment (adjacent to each other on the initial contour). Using the Potts model, frequent label jumps are penalized, but jumps are allowed at actual corners.

[0183] (twenty two),

[0184] Where σ is the smoothing penalty constant, which ensures that the generated wall lines are continuous straight line segments.

[0185] Finally adopted The algorithm iteratively minimizes the energy function described above. Based on the optimized graph cut result, elements belonging to the same label are grouped... The continuous set of points is fitted to form the final floor line.

[0186] The contour generated by this method closely matches the geometry of the actual wall surface, effectively correcting local deviations and missing wall lines in the initial contour. Figure 7 As shown.

[0187] The specific process for extracting the non-Manhattan ceiling line in step 7 is as follows:

[0188] Based on the two-dimensional floor profile obtained in step 6, and combined with the geometric assumption that the interior walls are vertical, the ceiling profile is calculated. To accommodate ceilings in non-Manhattan scenarios (such as sloping ceilings) or to improve the robustness of height estimation, this step does not simply rely on a fixed floor height, but introduces a planar model constraint of the ceiling point cloud:

[0189] First, perform ceiling plane fitting, using the ceiling point cloud clusters detected in step 3. The RANSAC algorithm is used to fit the optimal ceiling plane equation. Let the plane equation be... ,in It is the normal vector. This is the distance parameter. This parameter is solved iteratively by maximizing the number of interior points. If the fit fails or the point cloud is too small, the solution degenerates to using the global room height calculated in step 4. and ground height Calculate a fixed ceiling height .

[0190] Then use each ground contour point output in step 6 Find the intersection point between the vertical axis and the ceiling plane. If a planar equation model is used, the corresponding ceiling height is... The calculation formula is:

[0191] (twenty three),

[0192] The calculated height coordinates are combined with the planar coordinates to obtain the corresponding three-dimensional ceiling contour point set. This creates a ceiling line that is topologically consistent with the ground contour but highly accurate in matching the point cloud distribution.

[0193] In step 8, the specific process of generating the 3D vectorized layout is as follows:

[0194] The contour output in step 6 contains a dense set of points composed of numerous tiny line segments, which needs to be vectorized and simplified to remove collinear redundancy. This is done by traversing the ground contour point sequence and calculating the cosine of the angle between adjacent line segments. For three consecutive points... Construct vectors and Calculate the dot product of normalized vectors:

[0195] (twenty four),

[0196] If the included angle Less than the preset threshold (This invention takes) If the three points are collinear, then the point is considered collinear. Redundant points are removed; otherwise, they are retained. These are the key geometric corner points of the house. Repeat this process until the outline converges, resulting in a simplified set of corner points. Then, based on the ceiling plan model or height information from step 7, determine its corresponding ground vertex. and ceiling apex .connect and Generate a vertical line segment, which is the physical intersection line (wall edge) of two adjacent walls in three-dimensional space.

[0197] Finally, based on the topological connection order of the corner points, adjacent ground vertices are connected sequentially to form a closed ground outline, and adjacent ceiling vertices are connected to form a closed ceiling outline, ultimately forming a shape like... Figure 9 The image shows a closed, complete 3D vector model of an interior space that includes the geometry of the walls, floor, and ceiling.

[0198] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0199] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0200] This invention provides a method for predicting indoor layouts using multi-view panoramic images that integrates semantics and planarity. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for predicting indoor layout using multi-view panoramic images that integrates semantics and planarity, characterized in that, Includes the following steps: Step 1: Perform 3D semantic segmentation by combining 3D point cloud reconstruction with multi-view geometry and a large 2D segmentation model; Step 2: Combine semantic and prior information to filter out flying points and obtain a pure structured point cloud; Step 3, Multi-plane detection and classification; Plane detection is performed on the filtered point cloud. The multi-model random sampling consensus (RANSAC) algorithm is used, combined with semantic labels and normal vector constraints, to extract the wall plane, ceiling plane and ground plane respectively. Step 4, extract the initial contour: calculate the average height of the ground plane and the room height, project the detected ceiling point cloud vertically onto the ground plane, and fuse the projected ceiling point cloud, the original ground point cloud and the wall slice point cloud into an enhanced ground projection map; use morphological closing operation to connect the broken wall contours, fill the projection holes, extract the approximate geometric boundary of the room through the contour detection algorithm, and generate the initial layout contour. Step 5: Wall line generation based on plane fitting; For each detected wall point cloud cluster, perform three-dimensional plane fitting to obtain the wall's normal vector and center point. On the two-dimensional projection plane, use principal component analysis or line fitting to calculate the straight line equation of each wall. Calculate the intersection points of the straight line equations of adjacent or intersecting walls as candidate points for wall corners, and generate wall line segments with topological relationships. Step 6, 2D contour and wall-guided multi-label cut: Construct an energy function, use high-precision wall lines and initial contours, and generate a geometrically consistent ground and wall intersection contour by minimizing the energy function; Step 7, Non-Manhattan Ceiling Line Extraction: Introduce the geometric assumption that the wall is vertical and intersects the ceiling. Based on the vertical height of the room, calculate and generate the corresponding ceiling outline. Step 8: 3D vectorized layout generation; Based on the 3D planar outline and combined with the room's vertical height, the 2D wall lines are stretched along the vertical direction to generate the intersection lines of the wall and ceiling, as well as the intersection lines of the wall and floor, ultimately outputting a complete 3D vectorized interior layout model.

2. The method according to claim 1, characterized in that, Step 1 includes: First, load multi-view panoramic images from the HM3D dataset. Then, configure a spherical camera model to adapt to the panoramic field of view, taking into account the equidistant cylindrical projection characteristics of the panoramic images. Next, perform feature extraction and sparse reconstruction to generate an initial 3D point cloud that reflects the indoor geometric structure. Then, use the 2D segmentation large model SAM3 to extract multi-view image features and combine multi-view geometric mapping to obtain semantic labels for each 3D point, and identify doors, windows, walls, ceilings, floors and indoor furniture objects.

3. The method according to claim 2, characterized in that, Step 2 includes: Perform bidirectional filtering based on semantic tags to identify the point cloud corresponding to the door and window areas and remove noise points behind the doors and windows; Then, a statistical filtering algorithm is applied to the filtered structural point cloud. Let the neighborhood radius of the point cloud be r. The number of neighboring points within radius r for each point is counted, and isolated points with fewer than a threshold k are removed. The formula is as follows: （1）， in, To represent any point in the input point cloud, Given the input point cloud set, For point In radius The neighborhood point set within, This is a function for counting the number of points in the neighborhood. The threshold for the number of neighboring points; The final output is a pure structured point cloud. .

4. The method according to claim 3, characterized in that, Step 3 includes: First, use the semantic segmentation results to process the preprocessed point cloud. Divided into wall surfaces ,ground and ceiling Three subsets are used, and then an adaptive parameter adjustment mechanism is introduced for point clouds of different densities, by constructing... Tree calculation of local point cloud average spacing Dynamically set the RANSAC distance threshold With cluster radius Before RANSAC fitting, a density-based spatial clustering algorithm, DBSCAN, is introduced to project the wall point cloud onto... Planar clustering separates disconnected walls into independent clusters. Iterative RANSAC detection is performed within each cluster, where Indicates the k-th cluster; For any subset of point clouds, the planar model is defined by the normal vector. and distance parameters It is determined that the equation is satisfied: （2）， Where T represents the transpose; a, b, and c represent the three components of the plane normal vector n; and x, y, and z represent the coordinates of any point in space. A hypothetical model is generated by randomly sampling three interior points, and the points are calculated. Euclidean distance to the plane : （3）， if Then determine Let it be an interior point; Maximize the number of interior points through iteration To obtain the optimal planar model, points within the planar model are removed from the original point cloud. The remaining point cloud is then tested until the number of remaining points is insufficient or the preset maximum number of planar models is reached. During this process, geometric constraints are introduced to filter invalid planar models: for walls, the normal vector must be perpendicular to the surface. And the height span meets the requirements For the ground and ceiling, the normal vector is required to be approximately parallel to... axis; It is the verticality threshold; It is the minimum height threshold; Extract the set of all detected wall normal vectors Select the normal vector of the main wall surface with the most interior points. As a reference benchmark, for other wall surfaces Calculate the angle between wall surface j and the principal direction. : （4）， in Let m be the normal vector of the m-th wall. near Then, the Gram-Schmidt orthogonalization method is used to enforce constraints. orthogonal to .

5. The method according to claim 4, characterized in that, Step 4 includes: using the ceiling point cloud and ground point cloud detected by the RANSAC algorithm in step 3, and analyzing the Y coordinate of the point cloud to determine the highest point of the room. and lowest point The vertical distance between them is used to obtain the overall height of the room. : （5）， Define distance threshold : （6）， Based on distance threshold The point cloud of the entire room is divided into the following three parts: Ceiling: Points in the region near the minimum height at the Y-coordinate: （7）， Floor: Points in the region near the maximum height at the Y-coordinate: （8）， Other: Points located in the middle area; Obtain a more accurate ground reference: （9）， in This represents the estimated ground plane height. It retrieves the coordinates of 90% of the Y-coordinates of the ground point cloud set. It is the set of Y coordinates of the ground point cloud set; The average height of the ceiling is estimated using the median of the Y-coordinates of the ceiling point cloud: （10）， in This represents the estimated ceiling height. This is to obtain the coordinates of the median position of the ceiling point cloud set. It is the set of Y coordinates of the ceiling point cloud set; Using orthogonal projection, the ceiling point cloud is projected onto the ground plane: First, the Y coordinates of all copied ceiling points are uniformly set to the precise floor plane height. For any ceiling point Ceiling point New coordinates after projection onto the floor plane for: （11）， in , These are the horizontal coordinates of the original point. It is the Y-coordinate of the original point; The 3D projected point cloud is then converted into a binary image from a bird's-eye view. Morphological optimization is performed on the binary image using a rectangular structuring element to perform morphological closing operations. First, dilation is performed to connect the broken wall contours, and then erosion is performed to fill the voids. The formula is as follows: （12）， in It is the input boundary image. It is a structural element. Indicates expansion. Indicates corrosion; The contour retrieval mode and contour approximation method are used to extract the room boundary contours in the image, and small noise contours are removed by area filtering; and polygon fitting is performed on the extracted contours to transform jagged pixel edges into straight walls. Finally, the pixel coordinates on the image plane are... Convert to real-world 3D coordinates It outputs the initial layout outline, and the transformation logic is as follows: X-axis coordinate for: （13）， Z-axis coordinate for: （14）， Y-axis coordinate for: （15）， in, , These are the column coordinates and row coordinates of the image, respectively. It's resolution. It is the pixel value of the white space at the edge of the image. It is the minimum coordinate offset of the original point cloud in the X-axis direction. It is the minimum coordinate offset of the original point cloud in the Z-axis direction. It's the floor height.

6. The method according to claim 5, characterized in that, Step 5 includes: applying point cloud mapping to each wall surface. The wall point cloud is sliced ​​to extract points in the middle area of ​​the wall height. The slice range is: （16）， in It is a point cloud after slicing. It is the minimum height of the slice. is the highest height of the slice, and h is the height of the wall. Projecting the point cloud of the wall slices onto the XZ plane yields a two-dimensional point set. Then, PCA principal direction analysis is performed to obtain the covariance. : （17）， Where N represents the total number of points projected onto the XZ plane from the wall surface. It is the coordinate vector of the i-th point in the point set; right Perform eigenvalue decomposition The eigenvector corresponding to the largest eigenvalue The principal direction of the wall, and the vector perpendicular to the wall. It is a two-dimensional normal vector, then based on the center of the point set. Establish the two-dimensional straight line equation of the wall: （18）， in , , , , These are the components of the two-dimensional normal vector. It is a constant term calculated through the center point; For any two adjacent wall planes, a straight line and Solve the system of equations to find the intersection of the projected lines, which will be used as candidate corner points. The formula is as follows: （19）， If the determinant If there is an intersection, then the intersection point exists. Calculate the intersection point. to the original wall line endpoint Euclidean distance ;like Determine the location of the two walls at point Connecting forces the corresponding endpoints to the intersection. At the same time, update the wall line endpoints to This automatically closes the room outline; Det represents determinant, and snap is the preset distance judgment threshold.

7. The method according to claim 6, characterized in that, Step 6 includes: setting the initial two-dimensional contour point set As nodes in the graph, the set of wall lines optimized by PCA fitting is selected. As a set of candidate labels; for each contour point Assign an optimal wall label ;in This represents the nth contour point; This represents the straight line on the k-th wall surface; Constructing the energy function : （20）， in, It is the set of all adjacent pairs of points. They are two adjacent points on the initial two-dimensional contour. It is the specific wall label assigned to point p. It is the specific wall label assigned to point q. These are the planar coordinates of point p; data item Used to measure contour points Straight label to the assigned wall Geometric distance; label The equation of the straight line is Defined as: （21）， Smoothing terms Measuring neighboring nodes Consistency in label assignment; Using the Potts model, the frequent changes in the penalty label are penalized: （22）， Where σ is the smoothing penalty constant; Finally adopted The algorithm iteratively minimizes the energy function and, based on the optimized graph cut result, identifies elements belonging to the same label. The continuous set of points is fitted to form the final floor line.

8. The method according to claim 7, characterized in that, Step 7 includes: first, performing ceiling plane fitting, using the ceiling point cloud clusters detected in step 3. The RANSAC algorithm is used to fit the optimal ceiling plane equation. Let the plane equation be... ,in It is the normal vector. The distance parameter is obtained by iteratively solving for the distance parameter by maximizing the number of interior points. If the fitting fails or the point cloud is too small, the solution degenerates to using the global room height calculated in step 4. and ground height Calculate a fixed ceiling height ; Then use each ground contour point output in step 6 Find the intersection point of the ground contour point and the ceiling plane in the vertical direction. If a planar equation model is used, the corresponding ceiling height is... The calculation formula is: （23）， The calculated height coordinates are combined with the planar coordinates to obtain the corresponding three-dimensional ceiling contour point set. This forms a ceiling line that is topologically consistent with the ground contour but highly accurate in matching the point cloud distribution; It is the X-axis coordinate of the i-th ground contour point in the world coordinate system. is the Z-axis coordinate of the i-th ground contour point in the world coordinate system, and N represents the total number of discrete points that constitute the entire room's ground contour.

9. The method according to claim 8, characterized in that, Step 8 includes: By traversing the sequence of ground contour points, the cosine of the angle between adjacent line segments is calculated. For three consecutive points... Construct vectors sum vector Calculate the dot product of the normalized vectors: （24）， If the included angle Less than the preset threshold Then the three points are determined to be collinear. Redundant points are removed; otherwise, they are retained. As key geometric corner points of the house; repeat until the outline converges, resulting in a simplified set of corner points. ; It is the two-dimensional coordinate point of the corner point; Then, based on the ceiling plan model or height information from step 7, determine the corresponding ground vertex. and ceiling apex ;connect and Generate vertical line segments; Finally, based on the topological connection order of the corner points, adjacent ground vertices are connected in sequence to form a closed ground outline, and adjacent ceiling vertices are connected to form a closed ceiling outline, thus forming a complete three-dimensional vectorized layout model of the interior.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 9.

Images