A method, device and storage medium for three-dimensional modeling of a housing source

Abstract

The application provides a house source three-dimensional modeling method, equipment and a storage medium, wherein the method comprises using a camera to shoot a first video of the interior of a house, and splitting the first video into multiple indoor images. The indoor image set is subjected to semantic segmentation, and movable obstacles in the indoor image set are removed according to K labels. The active structure in the house interior structure image set is marked according to the K labels, and the two sides of the active structure are subjected to connected component analysis. The house interior structure image set is converted into M three-dimensional point clouds, and the M three-dimensional point clouds are reconstructed based on the spatial connection relationship. N reconstructed point clouds are subjected to Poisson reconstruction, and the interior three-dimensional model of the house source is obtained. The application adopts the semantic segmentation mode to exclude the influence of furniture and other movable obstacles on the house structure, uses the active structure to improve the accuracy of the spatial relationship between different regions in modeling, and represents the space separated by the active structure, so that the house is comprehensively and accurately three-dimensionally modeled.

Description

Technical Field

[0001] This invention relates to the field of data modeling technology, and in particular to a method, device and storage medium for three-dimensional modeling of housing. Background Technology

[0002] When choosing a property, homebuyers or renters should consider not only its location but also its internal structure to comprehensively assess its quality and expected lifespan. For example, a house may have multiple walls that separate adjacent areas to create different spaces, such as a living room, bedroom, bathroom, and kitchen.

[0003] Most existing technologies involve using cameras to capture images of different areas within different houses, converting these images into point cloud data, stitching together multiple point clouds, and then using the stitched point clouds to create a 3D model of the house. However, regardless of whether video is split into images or images are captured by a camera, the 3D house models generated by these methods have the following problems: due to the presence of movable structures, the spatial relationships between different areas are incorrectly modeled. For example, a movable door may be installed on one side of the master bedroom, allowing access to the bathroom when opened.

[0004] In conclusion, there is an urgent need for a method that can characterize the space divided by the active structure, thereby enabling comprehensive and accurate three-dimensional modeling of buildings. Summary of the Invention

[0005] To overcome the problems existing in related technologies, the purpose of this invention is to provide a method, device and storage medium for three-dimensional modeling of housing, wherein the method can characterize the space divided by the active structure, thereby comprehensively and accurately modeling the house in three dimensions.

[0006] A method for 3D modeling of housing, comprising:

[0007] The first video of the interior of the house is captured by a camera, and the first video is split into multiple interior images to obtain an interior image set;

[0008] Semantic segmentation is performed on the indoor image set to obtain K objects and K labels;

[0009] By removing movable obstacles from the indoor image set based on K labels, a set of images of the building's interior structure is obtained.

[0010] Based on the K labels, the active structures in the set of house interior structure images are marked, and the spatial connectivity relationship is obtained by performing connectivity analysis on both sides of the active structures.

[0011] The image set of the building's interior structure is converted into M three-dimensional point clouds. Based on the spatial connectivity relationship, the M three-dimensional point clouds are reconstructed to obtain N reconstructed point clouds; where N≤M.

[0012] Poisson reconstruction is performed on N reconstructed point clouds to obtain a 3D model of the interior of the property.

[0013] In a preferred embodiment of the present invention, the step of semantically segmenting the indoor image set to obtain K objects and K labels includes:

[0014] The indoor image set is input into the encoder of the semantic segmentation network, and dilated convolution is performed on the indoor image set to obtain low-level features and high-level features;

[0015] The low-level features are input into the decoder of the semantic segmentation network and convolved to obtain the first semantic features;

[0016] The high-level features are input into the dilated spatial pyramid pooling module of the encoder of the semantic segmentation network, and dilated convolutions are performed on the high-level features at different scales to obtain multi-scale features; the multi-scale features are then concatenated to obtain the second semantic features.

[0017] The first semantic feature and the second semantic feature are concatenated to obtain the concatenated feature.

[0018] The concatenated features are then convolved and upsampled sequentially to obtain K objects and K labels.

[0019] In a preferred embodiment of the present invention, before removing movable obstacles from the indoor image set based on K tags, the method further includes:

[0020] Read the interior floor plan of the house;

[0021] By combining the interior floor plan of the house with K tags, candidate areas for obstacles are selected.

[0022] In a preferred embodiment of the present invention, the step of removing movable obstacles from the indoor image set based on K labels to obtain the building interior structure image set includes:

[0023] Push or pull the structure to be detected within the candidate area of ​​the obstacle in different directions;

[0024] The structure to be detected is captured by a camera to obtain a second video;

[0025] Compare the positional relationship of the structure to be detected in the first video and the second video. If the overlapping area of ​​the structure to be detected in the first video and the second video is less than the first overlap threshold, then the structure to be detected is determined to be a movable obstacle.

[0026] Remove the objects and labels corresponding to the movable obstacles to obtain a set of images of the interior structure of the house.

[0027] In a preferred embodiment of the present invention, the step of marking the active structures in the set of house interior structure images according to K tags includes:

[0028] Compare the positional relationship of the structure to be detected in the first video and the second video. If the overlapping area of ​​the structure to be detected in the first video and the second video is greater than or equal to the first overlap threshold and less than the second overlap threshold, then the structure to be detected is determined to be an active structure.

[0029] In a preferred embodiment of the present invention, the step of performing connected component analysis based on both sides of the active structure to obtain spatial connectivity relationships includes:

[0030] Search for P indoor images in the indoor image set corresponding to the spaces on both sides of the active structure;

[0031] Detect the distribution of objects in the space on both sides of the active structure in P indoor images;

[0032] Based on the distribution of the objects, the spatial semantics on both sides of the active structure are determined to obtain the first keyword and the second keyword.

[0033] If the first keyword and the second keyword match, then it is determined that the spaces on both sides of the active structure are connected;

[0034] If the first keyword and the second keyword do not match, it is determined that the two sides of the active structure are spatially isolated.

[0035] In a preferred embodiment of the present invention, the step of converting the set of images of the interior structure of the house into M three-dimensional point clouds includes:

[0036] Obtain the camera intrinsic parameter matrix and depth scaling factor;

[0037] Input the camera intrinsic parameter matrix, depth scale factor, and image set of the building's internal structure into the Open3D library to generate M 3D point clouds.

[0038] In a preferred embodiment of the present invention, the step of reconstructing M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds includes:

[0039] If the spaces corresponding to the i-th 3D point cloud and the j-th 3D point cloud are connected, then data points are added between the i-th 3D point cloud and the j-th 3D point cloud to connect the i-th 3D point cloud and the j-th 3D point cloud.

[0040] If the i-th 3D point cloud and the j-th 3D point cloud are spatially isolated, then remove the data points between the i-th 3D point cloud and the j-th 3D point cloud to isolate the i-th 3D point cloud and the j-th 3D point cloud.

[0041] The present invention also provides a 3D modeling device for housing, characterized in that it comprises:

[0042] The video splitting module is used to capture a first video of the interior of a house using a camera, and then split the first video into multiple indoor images to obtain an indoor image set.

[0043] The semantic segmentation module is used to perform semantic segmentation on the indoor image set to obtain K objects and K labels;

[0044] A movable obstacle removal module is used to mark active structures in a set of images of the interior structure of a house according to K labels, and to perform connected component analysis on both sides of the active structures to obtain spatial connectivity relationships.

[0045] The point cloud reconstruction module is used to convert the set of images of the interior structure of the house into M three-dimensional point clouds, and reconstruct the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds; where N≤M;

[0046] The Poisson reconstruction module is used to perform Poisson reconstruction on N reconstructed point clouds to obtain a 3D model of the interior of the property.

[0047] The present invention also provides a storage medium, characterized in that the storage medium stores a computer program, which, when executed by one or more processors, implements a method for three-dimensional modeling of housing resources.

[0048] The beneficial effects of this invention are as follows:

[0049] The 3D modeling method for a house provided by this invention includes capturing a first video of the house's interior using a camera, splitting the first video into multiple interior images to obtain an interior image set. The interior images contain various objects, such as furniture, floors, walls, and beds. Semantic segmentation is used to distinguish different objects in each interior image, resulting in K objects and K labels. Semantic segmentation labels each pixel in the image with a corresponding semantic category, thereby achieving pixel-level scene reasoning. Based on the K labels, movable obstacles in the interior image set are removed, eliminating their impact and revealing the house's internal structure, resulting in an interior structure image set. Movable obstacles include furniture and everyday items; furniture includes coffee tables, bookshelves, and dining tables, while everyday items include delivery boxes and water cups. The K labels mark the active structures in the interior structure image set, and connectivity analysis is performed on both sides of the active structures to obtain spatial connectivity relationships. Active structures include movable doors and windows; movable doors can connect spaces on both sides, for example, a movable door can connect the master bedroom and the bathroom. Windows cannot connect spaces on either side. Spatial connectivity reflects the relationships between different spaces within a house; for example, the master bedroom and hallway are connected, the balcony and living room are connected, while the master bedroom and kitchen are isolated. The house's internal structure image set is converted into M 3D point clouds. Based on spatial connectivity, these M 3D point clouds are reconstructed to obtain N reconstructed point clouds, where N ≤ M. If two spaces are connected, data points are added between the corresponding two 3D point clouds to connect them. If two spaces are isolated, data points between the corresponding two 3D point clouds are removed to isolate them. Poisson reconstruction is performed on the N reconstructed point clouds to obtain the 3D model of the house's interior. This invention uses semantic segmentation to eliminate the influence of movable obstacles such as furniture on the house structure and utilizes active structures to improve the accuracy of modeling the spatial relationships between different areas, representing the spaces separated by active structures, thereby comprehensively and accurately modeling the house in 3D. Attached Figure Description

[0050] Figure 1 This is a flowchart of the three-dimensional modeling method for housing resources according to the present invention;

[0051] Figure 2 This is a flowchart of the connected component analysis based on both sides of the active structure according to the present invention;

[0052] Figure 3 This is a structural diagram of the semantic segmentation network of the present invention. Detailed Implementation

[0053] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Example 1

[0054] like Figure 1 As shown, this embodiment provides a method for 3D modeling of housing, including:

[0055] S1: Use a camera to capture the first video of the interior of the house, and split the first video into multiple interior images to obtain an interior image set;

[0056] S2: Perform semantic segmentation on the indoor image set to obtain K objects and K labels;

[0057] S3: Remove movable obstacles from the indoor image set based on K labels to obtain the image set of the house's internal structure;

[0058] S4: Mark the active structures in the set of house interior structure images based on K labels, and perform connected component analysis on both sides of the active structures to obtain spatial connectivity relationships;

[0059] S5: Convert the set of images of the interior structure of the house into M three-dimensional point clouds, and reconstruct the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds; where N≤M;

[0060] S6: Perform Poisson reconstruction on N reconstructed point clouds to obtain a 3D model of the interior of the property.

[0061] The first video consists of multiple indoor images. The first video is split into segments according to time sequence, so that the first indoor image to the last indoor image has temporal continuity.

[0062] The semantic segmentation of the indoor image set to obtain K objects and K labels includes:

[0063] S21: Input the indoor image set into the encoder of the semantic segmentation network, perform dilated convolution on the indoor image set to obtain low-level features and high-level features;

[0064] S22: Input the low-level features into the decoder of the semantic segmentation network and perform convolution to obtain the first semantic features;

[0065] S23: Input the high-level features into the dilated spatial pyramid pooling module of the encoder of the semantic segmentation network, perform dilated convolution on the high-level features at different scales to obtain multi-scale features; concatenate the multi-scale features to obtain the second semantic features;

[0066] S24: Concatenate the first semantic feature and the second semantic feature to obtain the concatenated feature;

[0067] S25: Perform convolution and upsampling on the concatenated features in sequence to obtain K objects and K labels.

[0068] Semantic segmentation is a pixel-level classification task in computer vision, aiming to label each pixel in an image with its corresponding semantic category. The DeepLab algorithm combines dilated convolution and spatial pyramid pooling to enhance multi-scale feature extraction capabilities. The DeepLab algorithm is characterized by high edge detection accuracy and is suitable for complex scenes.

[0069] Semantic segmentation networks consist of an encoder and a decoder. The encoder compresses and encodes indoor images to obtain coded features. The decoder upsamples these coded features to the desired resolution. Combined, the encoder and decoder deliver expressive high-level information throughout the semantic segmentation network. Figure 3 As shown, the encoder includes a dilated convolution module and a dilated spatial pyramid pooling module. The first output of the dilated convolution module is connected to the input of the dilated spatial pyramid pooling module, and the second output of the dilated convolution module is connected to the input of the decoder.

[0070] Dilated convolution inserts zeros into the convolution kernel to increase its size without increasing the number of learnable parameters. Dilated convolution offers the following advantages: (1) it can extract denser features at deeper levels; (2) it allows for rate-controlled receptive fields; and (3) it retains the same number of learnable parameters as regular convolution. Using dilated convolution to build deeper networks allows for the retention of more advanced information at a finer resolution without increasing the number of parameters.

[0071] The dilated convolutional spatial pyramid pooling module resamples features of indoor images at multiple scales, then performs pooling on these resampled features. For example, it decomposes high-level features into different scales, obtaining multiple high-order features at different scales. All high-order features are then averaged to obtain the second semantic features. For instance, the high-level features undergo five different operations in the dilated spatial pyramid pooling module to produce five outputs: one 1×1 convolution, three dilated convolutions with different rates, and one global average pooling. These five outputs are then passed through a connection layer and a 1×1 convolutional layer to obtain the second semantic features.

[0072] A 1×1 convolution is performed on the first semantic feature, and the second semantic feature is upsampled by 4 times. The first semantic feature after the 1×1 convolution and the upsampled second semantic feature are then concatenated to obtain the concatenated feature. The concatenated feature is then concatenated by 3×3 convolution and upsampled by 4 times to obtain the prediction result, which includes K objects and K labels.

[0073] The spatial connectivity relationship is obtained by performing connectivity analysis on both sides of the active structure, including:

[0074] S42: Search for P indoor images in the indoor image set corresponding to the space on both sides of the active structure;

[0075] S43: Detect the distribution of objects in the space on both sides of the active structure in P indoor images;

[0076] S44: Determine the spatial semantics of both sides of the active structure based on the distribution of the objects to obtain the first keyword and the second keyword;

[0077] S45: If the first keyword and the second keyword match, then it is determined that the spaces on both sides of the active structure are connected;

[0078] S46: If the first keyword and the second keyword do not match, it is determined that the two sides of the active structure are spatially isolated.

[0079] Before step S42, there is also step S41: marking the active structures in the house interior structure image set according to K labels.

[0080] The spaces on either side of a movable structure may be connected, such as a sliding door between the master bedroom and the bathroom. Pulling the door horizontally opens or closes the passage between the two spaces. Alternatively, the spaces on either side of a movable structure may be isolated, such as a window leading to the kitchen and outside the window. Whether the window is open or closed, the kitchen and the outside are isolated, and people and objects cannot pass through the window from the kitchen to the outside. Therefore, it is necessary to identify the spaces on either side of the movable structure from P interior images and analyze the semantics of these spaces. For example, if the movable structure is a sliding door, the space on the first side includes a bed and a television, while the space on the second side includes a toilet and a sink. Based on the object distribution on the first side of the movable door, the primary keyword is "master bedroom." Based on the spatial distribution on the second side, the secondary keyword is "bathroom." Since the keywords "master bedroom" and "bathroom" match, the spaces on either side of the movable door are determined to be connected.

[0081] Poisson reconstruction is a 3D surface reconstruction algorithm based on implicit functions. It generates a closed triangular mesh model from a point cloud with normal vectors by solving the Poisson equation. This is achieved through implicit functions. Distinguish between the inner and outer spaces of object M. The external space corresponding to object M. This corresponds to the internal space of object M. The Poisson equation is constructed using the point cloud normal field V. The implicit function is obtained by solving this equation. Then, the isosurface is extracted. Based on the Gaussian divergence theorem, the gradient of the smoothed implicit function is equal to the vector field of the normal field. The implicit function is solved by discretizing the octree space and using least squares approximation. .

[0082] The 3D modeling method for a property provided in this embodiment includes capturing a first video of the interior of the house using a camera, splitting the first video into multiple interior images to obtain an interior image set. The interior images contain various objects, such as furniture, floors, walls, and beds. Semantic segmentation is used to distinguish different objects in each interior image, resulting in K objects and K labels. Semantic segmentation labels each pixel in the image with a corresponding semantic category, thereby achieving pixel-level scene reasoning. Movable obstacles in the interior image set are removed based on the K labels, eliminating their impact and revealing the internal structure of the house, resulting in an interior structure image set. Movable obstacles include furniture and everyday items; furniture includes coffee tables, bookshelves, and dining tables, while everyday items include delivery boxes and water cups. Active structures in the interior structure image set are marked using the K labels, and connected component analysis is performed on both sides of the active structures to obtain spatial connectivity relationships. Active structures include movable doors and windows; movable doors can connect spaces on both sides, for example, a movable door can connect the master bedroom and the bathroom. Windows cannot connect spaces on either side. Spatial connectivity reflects the relationships between different spaces within a house; for example, the master bedroom and hallway are connected, the balcony and living room are connected, while the master bedroom and kitchen are isolated. The house's internal structure image set is converted into M 3D point clouds. Based on spatial connectivity, these M 3D point clouds are reconstructed to obtain N reconstructed point clouds, where N ≤ M. If two spaces are connected, data points are added between the corresponding two 3D point clouds to connect them. If two spaces are isolated, data points between the corresponding two 3D point clouds are removed to isolate them. Poisson reconstruction is performed on the N reconstructed point clouds to obtain the 3D model of the house's interior. This invention uses semantic segmentation to eliminate the influence of movable obstacles such as furniture on the house structure and utilizes active structures to improve the accuracy of modeling the spatial relationships between different areas, representing the spaces separated by active structures, thereby comprehensively and accurately modeling the house in 3D.

[0083] Example 2

[0084] like Figure 1As shown, this embodiment provides a method for 3D modeling of housing. This embodiment differs from Embodiment 1 in that it includes the following methods:

[0085] S1: Use a camera to capture the first video of the interior of the house, and split the first video into multiple interior images to obtain an interior image set;

[0086] S2: Perform semantic segmentation on the indoor image set to obtain K objects and K labels;

[0087] S3: Remove movable obstacles from the indoor image set based on K labels to obtain the image set of the house's internal structure;

[0088] S4: Mark the active structures in the set of house interior structure images based on K labels, and perform connected component analysis on both sides of the active structures to obtain spatial connectivity relationships;

[0089] S5: Convert the set of images of the interior structure of the house into M three-dimensional point clouds, and reconstruct the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds; where N≤M;

[0090] S6: Perform Poisson reconstruction on N reconstructed point clouds to obtain a 3D model of the interior of the property.

[0091] Before removing movable obstacles from the indoor image set based on K labels, the method further includes:

[0092] S21': Read the interior floor plan of the house;

[0093] S22': Combine the interior floor plan of the house with K labels to filter out candidate areas for obstacles.

[0094] The interior floor plan depicts the overall structure of the house, with dimensions of each part marked according to a scale. For example, load-bearing walls are represented by solid black lines with a thickness of 240mm, while non-load-bearing walls are represented by double or hollow lines with a thickness of 120mm. The unit of measurement on the interior floor plan is millimeters, and the scale is 1:100, meaning 1 centimeter on the drawing equals 1 meter in reality. For example, based on the interior floor plan and K labels, the locations of the living room and secondary bedroom can be found. A partition wall separates the living room and secondary bedroom, and this partition wall is a potential obstacle area.

[0095] The step of removing movable obstacles from the indoor image set based on K labels to obtain the building interior structure image set includes:

[0096] S31: Push or pull the structure to be detected within the candidate area of ​​the obstacle in different directions;

[0097] S32: Use a camera to capture the structure to be detected to obtain a second video;

[0098] S33: Compare the positional relationship of the structure to be detected in the first video and the second video. If the overlapping area of ​​the structure to be detected in the first video and the second video is less than the first overlap threshold, then the structure to be detected is determined to be a movable obstacle.

[0099] S34: Remove the objects and labels corresponding to the movable obstacles to obtain a set of images of the interior structure of the house.

[0100] Objects in the obstacle candidate area are either movable obstacles or moving structures. Movable obstacles include furniture such as sofas and coffee tables, while moving structures include sliding doors and windows. The object to be detected is pushed or pulled horizontally, vertically, and at a 45-degree angle. The same object is shown in both the first and second videos. The positional relationship between the objects in the first and second videos is compared, and the type of object to be detected is determined based on the degree of displacement of each object.

[0101] If the overlapping area of ​​the structure to be detected in the first video and the second video is less than a first overlap threshold, meaning the range of motion and movable distance of the structure to be detected are large, then the structure to be detected is determined to be a movable obstacle. If the overlapping area of ​​the structure to be detected in the first video and the second video is greater than or equal to the first overlap threshold and less than a second overlap threshold, meaning the range of motion and movable distance of the structure to be detected are small, then the structure to be detected is determined to be a movable structure. For example, a sofa can be moved from one corner of the living room to another. Due to the limiting effect of the sliding rail on the movable door, the movable door can only move from the far left to the far right. Therefore, the range of motion and movable distance of the movable door are both smaller than that of the sofa. The movable door is a movable structure, and the sofa is a movable obstacle.

[0102] In this embodiment, the structure to be detected is pushed or pulled, and the displacement of the structure is detected by comparing the first video and the second video. The structure to be detected with a large displacement is determined to be a movable obstacle, and the structure to be detected with a small displacement is determined to be an active structure.

[0103] Example 3

[0104] like Figure 1 As shown, this embodiment provides a method for 3D modeling of housing. This embodiment differs from Embodiment 1 in that it includes the following methods:

[0105] S1: Use a camera to capture the first video of the interior of the house, and split the first video into multiple interior images to obtain an interior image set;

[0106] S2: Perform semantic segmentation on the indoor image set to obtain K objects and K labels;

[0107] S3: Remove movable obstacles from the indoor image set based on K labels to obtain the image set of the house's internal structure;

[0108] S4: Mark the active structures in the set of house interior structure images based on K labels, and perform connected component analysis on both sides of the active structures to obtain spatial connectivity relationships;

[0109] S5: Convert the set of images of the interior structure of the house into M three-dimensional point clouds, and reconstruct the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds; where N≤M;

[0110] S6: Perform Poisson reconstruction on N reconstructed point clouds to obtain a 3D model of the interior of the property.

[0111] The process of converting the set of images of the building's interior structure into M three-dimensional point clouds includes:

[0112] S51: Obtain the camera intrinsic parameter matrix and depth scaling factor;

[0113] S52: Input the camera intrinsic parameter matrix, depth scale factor, and image set of the building's internal structure into the Open3D library to generate M 3D point clouds.

[0114] Camera intrinsics, depth scaling factor, RGB images of the house's interior structure, and depth images of the house's interior structure are input into Open3D to generate M 3D point clouds. Camera intrinsics include focal length and principal point. Open3D is an open-source 3D data processing library that supports the processing and visualization of point clouds, meshes, and RGBD images. Preferably, the RGB image of the house's interior structure is first downsampled by a factor of 4 to improve computational efficiency, resulting in a downsampled RGB image. Open3D is then used to fuse the downsampled RGB image and the depth image of the house's interior structure into a 3D point cloud. A 3D point cloud is a collection of points in 3D space. This embodiment uses semantic segmentation to generate 3D point clouds for different house interior spaces, with one 3D point cloud corresponding to one house interior space, such as the master bedroom. The M 3D point clouds correspond to M house interior spaces, with some 3D point clouds connected and others isolated.

[0115] The process of reconstructing M 3D point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds includes:

[0116] S53: If the spaces corresponding to the i-th 3D point cloud and the j-th 3D point cloud are connected, then data points are added between the i-th 3D point cloud and the j-th 3D point cloud to connect the i-th 3D point cloud and the j-th 3D point cloud.

[0117] S54: If the i-th 3D point cloud and the j-th 3D point cloud are spatially isolated, then remove the data points between the i-th 3D point cloud and the j-th 3D point cloud to isolate the i-th 3D point cloud and the j-th 3D point cloud.

[0118] As an example, the second 3D point cloud corresponds to the master bedroom, and the fourth 3D point cloud corresponds to the bathroom. The master bedroom and bathroom are connected by a sliding door. Therefore, 'temp1' data points are added between the second and fourth 3D point clouds. These 'temp1' data points are distributed along the path from the second 3D point cloud to the fourth 3D point cloud, thus connecting the second and fourth 3D point clouds. The third 3D point cloud corresponds to the master bedroom, and the fifth 3D point cloud corresponds to the living room. The master bedroom and living room are isolated; in this invention, isolation means they are not directly connected, for example, they are connected by a corridor. 'temp2' data points are scattered between the third and fifth 3D point clouds, causing them to be approximately connected. These 'temp2' data points are removed to isolate the third and fifth 3D point clouds.

[0119] This embodiment also provides a 3D modeling device for housing, including:

[0120] The video splitting module is used to capture a first video of the interior of a house using a camera, and then split the first video into multiple indoor images to obtain an indoor image set.

[0121] The semantic segmentation module is used to perform semantic segmentation on the indoor image set to obtain K objects and K labels;

[0122] A movable obstacle removal module is used to mark active structures in a set of images of the interior structure of a house according to K labels, and to perform connected component analysis on both sides of the active structures to obtain spatial connectivity relationships.

[0123] The point cloud reconstruction module is used to convert the set of images of the interior structure of the house into M three-dimensional point clouds, and reconstruct the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds; where N≤M;

[0124] The Poisson reconstruction module is used to perform Poisson reconstruction on N reconstructed point clouds to obtain a 3D model of the interior of the property.

[0125] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the 3D modeling method for housing resources described in any one of embodiments 1-3. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0126] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0127] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for 3D modeling of housing, characterized in that, include: The first video of the interior of the house is captured by a camera, and the first video is split into multiple interior images to obtain an interior image set; Semantic segmentation is performed on the indoor image set to obtain K objects and K labels; The semantic segmentation of the indoor image set to obtain K objects and K labels includes: The indoor image set is input into the encoder of the semantic segmentation network, and dilated convolution is performed on the indoor image set to obtain low-level features and high-level features; The low-level features are input into the decoder of the semantic segmentation network and convolved to obtain the first semantic features; The high-level features are input into the dilated spatial pyramid pooling module of the encoder of the semantic segmentation network, and dilated convolutions are performed on the high-level features at different scales to obtain multi-scale features; the multi-scale features are then concatenated to obtain the second semantic features. The first semantic feature and the second semantic feature are concatenated to obtain the concatenated feature. The concatenated features are sequentially convolved and upsampled to obtain K objects and K labels; By removing movable obstacles from the indoor image set based on K labels, a set of images of the building's interior structure is obtained. Based on the K labels, the active structures in the set of house interior structure images are marked, and the spatial connectivity relationship is obtained by performing connectivity analysis on both sides of the active structures. The spatial connectivity relationship is obtained by performing connectivity analysis on both sides of the active structure, including: Search for P indoor images in the indoor image set corresponding to the spaces on both sides of the active structure; Detect the distribution of objects in the space on both sides of the active structure in P indoor images; Based on the distribution of the objects, the spatial semantics on both sides of the active structure are determined to obtain the first keyword and the second keyword. If the first keyword and the second keyword match, then it is determined that the spaces on both sides of the active structure are connected; If the first keyword and the second keyword do not match, it is determined that the two sides of the active structure are spatially isolated. The image set of the building's interior structure is converted into M three-dimensional point clouds. Based on the spatial connectivity relationship, the M three-dimensional point clouds are reconstructed to obtain N reconstructed point clouds; where N≤M. The process of reconstructing M 3D point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds includes: If the spaces corresponding to the i-th 3D point cloud and the j-th 3D point cloud are connected, then data points are added between the i-th 3D point cloud and the j-th 3D point cloud to connect the i-th 3D point cloud and the j-th 3D point cloud. If the spatial isolation between the i-th 3D point cloud and the j-th 3D point cloud is achieved, then the data points between the i-th 3D point cloud and the j-th 3D point cloud are removed to isolate the i-th 3D point cloud and the j-th 3D point cloud. Poisson reconstruction is performed on N reconstructed point clouds to obtain a 3D model of the interior of the property.

2. The method for three-dimensional modeling of housing resources according to claim 1, characterized in that, Before removing movable obstacles from the indoor image set based on K labels, the method further includes: Read the interior floor plan of the house; By combining the interior floor plan of the house with K tags, candidate areas for obstacles are selected.

3. The method for three-dimensional modeling of housing resources according to claim 2, characterized in that, The step of removing movable obstacles from the indoor image set based on K labels to obtain the building interior structure image set includes: Push or pull the structure to be detected within the candidate area of ​​the obstacle in different directions; The structure to be detected is captured by a camera to obtain a second video; Compare the positional relationship of the structure to be detected in the first video and the second video. If the overlapping area of ​​the structure to be detected in the first video and the second video is less than the first overlap threshold, then the structure to be detected is determined to be a movable obstacle. Remove the objects and labels corresponding to the movable obstacles to obtain a set of images of the interior structure of the house.

4. The method for three-dimensional modeling of housing resources according to claim 3, characterized in that, The step of marking the active structures in the set of house interior structure images based on K labels includes: Compare the positional relationship of the structure to be detected in the first video and the second video. If the overlapping area of ​​the structure to be detected in the first video and the second video is greater than or equal to the first overlap threshold and less than the second overlap threshold, then the structure to be detected is determined to be an active structure.

5. The method for three-dimensional modeling of housing resources according to claim 1, characterized in that, The process of converting the set of images of the building's interior structure into M three-dimensional point clouds includes: Obtain the camera intrinsic parameter matrix and depth scaling factor; Input the camera intrinsic parameter matrix, depth scale factor, and image set of the building's internal structure into the Open3D library to generate M 3D point clouds.

6. A 3D modeling device for housing, characterized in that, include: The video splitting module is used to capture a first video of the interior of a house using a camera, and then split the first video into multiple indoor images to obtain an indoor image set. A semantic segmentation module is used to perform semantic segmentation on the indoor image set to obtain K objects and K labels. The process of performing semantic segmentation on the indoor image set to obtain K objects and K labels includes: inputting the indoor image set into the encoder of the semantic segmentation network, performing dilated convolution on the indoor image set to obtain low-level features and high-level features; inputting the low-level features into the decoder of the semantic segmentation network for convolution to obtain first semantic features; inputting the high-level features into the dilated spatial pyramid pooling module of the encoder of the semantic segmentation network, performing dilated convolution on the high-level features at different scales to obtain multi-scale features; concatenating the multi-scale features to obtain second semantic features; concatenating the first semantic features and the second semantic features to obtain concatenated features; and sequentially convolving and upsampling the concatenated features to obtain K objects and K labels. A movable obstacle removal module is used to mark active structures in a set of interior building structure images based on K labels, and to perform connected component analysis on both sides of the active structures to obtain spatial connectivity. The connected component analysis on both sides of the active structures to obtain spatial connectivity includes: searching for P indoor images in the indoor image set corresponding to the spaces on both sides of the active structures; detecting the distribution of objects in the spaces on both sides of the active structures in the P indoor images; determining the spatial semantics on both sides of the active structures based on the object distribution to obtain a first keyword and a second keyword; if the first keyword and the second keyword match, then the spaces on both sides of the active structures are determined to be connected; if the first keyword and the second keyword do not match, then the spaces on both sides of the active structures are determined to be isolated. A point cloud reconstruction module is used to convert a set of images of the interior structure of a building into M three-dimensional point clouds, and to reconstruct the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds; where N≤M; the step of reconstructing the M three-dimensional point clouds based on the spatial connectivity relationship to obtain N reconstructed point clouds includes: if the spatial connection between the i-th and j-th three-dimensional point clouds is such that data points are added between the i-th and j-th three-dimensional point clouds to connect them; if the spatial connection between the i-th and j-th three-dimensional point clouds is such that data points are removed between them to isolate them. The Poisson reconstruction module is used to perform Poisson reconstruction on N reconstructed point clouds to obtain a 3D model of the interior of the property.

7. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by one or more processors, implements the three-dimensional modeling method for housing resources as described in any one of claims 1-5.

Images