A method for indoor scene understanding and structured modeling based on multi-source data fusion

Abstract

The present invention discloses an indoor scene understanding and structured modeling method based on multi-source data fusion, the modeling method includes: indoor scene multi-source data fusion with joint key point detection and description; multi-granularity understanding of indoor scenes based on semantic enhancement and multi-task collaboration; fusion data-driven and model-driven reconstruction of indoor full-element three-dimensional structured models. A set of robust and robust indoor automated modeling methods is established to provide important technical support for indoor application services.

Description

Technical Field

[0001] The present invention relates to the field of indoor scene understanding and three-dimensional modeling, and in particular to an indoor scene understanding and structured modeling method based on multi-source data fusion. Background Art

[0002] Accurate, detailed, and structured indoor 3D models are an important part of the construction of a real 3D China. Compared with 2D maps, indoor 3D models not only have higher geometric accuracy and visual realism, but also provide richer scene knowledge and semantic information. They are widely used in robot navigation, emergency services, architectural design, virtual reality and other fields. Therefore, the core of indoor 3D model reconstruction lies in the semantic understanding of indoor scenes and the reconstruction of structured models.

[0003] In recent years, with the development of 3D SLAM technology, indoor mobile measurement systems have integrated IMU, laser scanners, digital cameras and other instruments and equipment. They can continuously obtain two-dimensional images and three-dimensional laser point clouds in scenes without GNSS, reducing the impact caused by occlusion. They have significant advantages in information collection of large-scale indoor scenes, making semantic understanding and rapid model reconstruction of large-scale indoor scenes possible. However, due to the differences in the representation form, data structure, spatial reference and other aspects of multi-source data such as point clouds and images, as well as the complex structure of indoor scenes, occlusion and similarity between target entities, they have brought great challenges to the semantic understanding of indoor scenes and the automatic reconstruction of three-dimensional models. Specifically, they are: difficult to fuse multi-source data; difficult to understand complex indoor scenes; low degree of automation of indoor modeling, low model accuracy and other problems. Summary of the invention

[0004] In view of the above problems, the present invention proposes an indoor scene understanding and structured modeling method based on multi-source data fusion in order to provide a method to overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present invention, a method for indoor scene understanding and structured modeling based on multi-source data fusion is provided, and the reconstruction method comprises:

[0006] Multi-source data fusion of indoor scenes with joint key point detection and description;

[0007] Multi-granularity understanding of indoor scenes based on semantic enhancement and multi-task collaboration;

[0008] Integrate data-driven and model-driven approaches to reconstruct a three-dimensional structured model of all indoor elements.

[0009] Optionally, the indoor scene multi-source data fusion of the joint key point detection and description specifically includes:

[0010] (1) Calculation of local feature descriptors based on the ultra-wide receptive field mechanism, including: for 2D images, using dilated convolution with a gradually increasing dilation rate to achieve an ultra-wide receptive field mechanism, and finally generating feature maps and score maps with the same resolution as the original image;

[0011] For 3D point clouds, the same strategy is used to improve the core point convolutional network to learn the features of each 3D point, and finally output the high-dimensional feature vector and saliency of each point;

[0012] The high-dimensional feature vector after L2 regularization is directly compared between the point cloud and the image, and the cosine similarity is used to establish the corresponding relationship of the same name;

[0013] During training, it is expected that the feature values ​​of pixels and point pairs with the same name will be adjusted to be as similar as possible;

[0014] (2) Keypoint detection based on the saliency of the coordinate space and feature space, including: selecting the most significant feature dimension and determining the local maximum of the feature dimension in the spatial neighborhood R;

[0015] One dot / pixel To become a key point, two conditions must be met: saliency in feature space and saliency in coordinate space;

[0016] (3) A coarse-to-fine optimization strategy that takes into account both key point detection and description, including:

[0017] In one forward propagation, the detection and description of 2D and 3D key points are jointly optimized, and a coarse-to-fine loss function that takes into account both key point detection and description is designed, as shown in formula (1):

[0018] (1)

[0019] in, Relatively coarse supervision of each descriptor, Focus on supervising descriptors that are prone to mismatching, is the scale factor;

[0020] In order to improve the convergence of the feature description learning process, the circle loss is used as the feature descriptor loss function. The key points are still repeatable and highly distinguishable when the viewpoint and light change. The loss function is designed to encourage points with high distinguishability to have significant scores.

[0021] The high-precision cross-modal data fusion method of joint key point detection and description realizes the fusion of multimodal image and point cloud data under a unified network framework to form an enhanced point cloud, providing data guarantee for indoor scene understanding and three-dimensional model reconstruction.

[0022] Optionally, the multi-granularity understanding of indoor scenes based on semantic enhancement and multi-task collaboration specifically includes:

[0023] In order to solve the problem of over-segmentation of indoor spaces with multiple houses and complex connections, a single-house segmentation method integrating semantic constraints and multi-labeled graph cuts is proposed to segment the unordered point cloud into single rooms with semantic information, thus achieving coarse-grained understanding of indoor scenes.

[0024] According to the component-level objects of a single house after segmentation, based on the fused multi-source data, the network generalization ability is improved by utilizing task collaboration, inductive transfer, and shared representation to achieve multi-level and fine-grained indoor scene understanding, where the multi-levels include: positioning, semantics, and shape; the fine-grained includes component-level instance segmentation.

[0025] Optionally, the single house segmentation method integrating semantic constraints and multi-labeled graph cuts specifically includes:

[0026] Perform visual analysis on sampled moving trajectory points, including:

[0027] The visible point cloud range is limited based on the position of the door, and the moving trajectory points are segmented into subspaces to form the initial division of the space;

[0028] Based on the visual point cloud similarity and spatial smoothness of adjacent trajectory points, a multi-labeled graph cut energy function is constructed to automatically cluster similar visual point clouds and complete the segmentation of a single house.

[0029] (2)

[0030] Data items: ,in It is a trajectory point Assigned track segment The penalty value is With track segment The greater the visible point cloud coverage, the penalty value of the energy function The smaller it is, the trajectory points tend to cluster in trajectory segments with greater similarity;

[0031] Smoothing term: , where the component It is used to regularize labels and penalize adjacent trajectory points that are assigned different label values. If adjacent trajectory points belong to the same space, the smoothing term is 0. If adjacent trajectory points do not belong to the same space, the greater the coverage of two adjacent trajectory points and the smaller the point spacing, the greater the penalty value of the smoothing term, which is close to 1. The smoothing term reduces the redundant subspace, so that adjacent trajectory points are clustered into the same house.

[0032] Multi-level and fine-grained understanding of indoor scenes combined with multi-task deep learning, including:

[0033] According to each house after segmentation, a backbone feature learning network based on graph neural network and structure-attention mechanism is developed to analyze the neighborhood relationship composition and structural information in three-dimensional space;

[0034] Based on the feature learning backbone network, a multi-task deep learning network integrating 3D object detection, semantic segmentation, and instance segmentation is developed to achieve multi-level and fine-grained indoor scene understanding under a unified network framework.

[0035] Optionally, the multi-level and fine-grained understanding of indoor scenes combined with multi-task deep learning specifically includes:

[0036] The backbone feature learning network based on graph neural network and structure-attention mechanism includes:

[0037] First, for the original large-scale point set, we use a combination of traditional random sampling, spherical sampling, regional sampling and improved autoencoders to extract a subset of the scene and reduce the number of elements in the point set.

[0038] Then, a graph convolutional network is used to connect the relationships between points in the global or local range on the streamlined subset;

[0039] Finally, for the primary feature expression completed by graph convolution processing, a deep feature extraction skeleton connected by a hierarchical convolutional neural network is constructed, and the attention mechanism is introduced in the dimension of the correlation of the structure in space to form a three-dimensional object cognition process that conforms to the human brain's cognition, improve the cognitive ability layer by layer, complete the feature extraction of the entire scene, and form the construction of the relationship between classes within the scene;

[0040] The object detection branch based on convolutional attention mechanism and non-local feature pyramid includes:

[0041] Integrate spatial attention and channel attention into the implementation of point cloud convolution to learn feature information that is more beneficial to 3D object detection;

[0042] Use non-local feature pyramids to fuse information based on the global similarity of features, extract deformable features of 3D objects of different shapes and scales, and predict 3D bounding boxes; construct a Gaussian distribution heat map of irregular shape point clouds based on the center point of the 3D bounding box;

[0043] The semantic segmentation branch based on structural primitive relationship combination reasoning includes:

[0044] The accuracy of semantic segmentation depends greatly on the combination relationship between spatial structure primitives and the retention of point-by-point positioning information. The semantic segmentation method includes: spatial structure primitive expression, relationship reasoning and combination generalization and point-by-point spatial information recovery.

[0045] The instance segmentation branch that takes into account semantic boundary information and target location information includes:

[0046] By utilizing the localization information of the target detection branch method and the category information of the semantic segmentation branch, the instance segmentation problem is transformed into two coupled processes: confidence matrix prediction and boundary localization mask prediction.

[0047] Optionally, the fusion of data-driven and model-driven reconstruction of a three-dimensional structured model of all indoor elements specifically includes:

[0048] Data-driven reconstruction of indoor house models based on line and surface mixed constraints;

[0049] Fitting indoor instance segmentation object model based on model-driven;

[0050] Construct a structural relationship diagram of the indoor three-dimensional structured model.

[0051] Optionally, the data-driven reconstruction of the indoor house model based on line and surface mixed constraints specifically includes:

[0052] According to the segmented single house, horizontal slices are made at a certain height from the ceiling, the point cloud of the horizontal slices is converted into a binary image, and a line segment detector is used to extract straight lines from the binary image;

[0053] Based on the image polygon block optimization theory and G2o solver, the angle and distance deviation of the line segment are corrected globally.

[0054] After the global correction of the line segment error term, the indoor scene is still expressed by independent small line segments with different labels, and the redundancy is too large. The region growing algorithm is used to cluster the line segments with similar features and determine the main line segments;

[0055] Combine the geometric structure information of the main line segment with the semantic information of the segmented single house to complete the structural line of the model and reconstruct the indoor 3D model with correct topological relationship;

[0056] Extending the main line segments forms a closed two-dimensional plane graph, and the lines in the plane graph have a topological structure;

[0057] Project the segmented single house point cloud onto a two-dimensional polygonal plane, and assign labels to each grid based on the projected point cloud. , including outdoor tags ;

[0058] Each main line segment is divided into a one-dimensional uniform grid. The one-dimensional grid label vector is determined by the projected small line segment. The label set is ,Line segments with labels are used to separate grids belonging to different houses;

[0059] The energy function is minimized by multi-labeled graph cuts, and the two-dimensional polygons and line segments are globally optimized to construct the plane model. The three-dimensional structured model is reconstructed according to the elevation information of the ceiling and floor planes.

[0060] Optionally, the model-driven fitting of the indoor instance segmentation object model specifically includes:

[0061] According to the indoor instance segmentation objects, 3DMax is used to build a template library, and an automatic registration method between the instance segmentation point cloud and the template library model is proposed to combine the indoor objects and room models to generate a full-element 3D model of the indoor environment.

[0062] The indoor full-element three-dimensional model is used as source data, and the instance segmentation point cloud is used as target data;

[0063] Object-level key points are defined as the two endpoints of the intersection line of adjacent faces;

[0064] Extract object-level key points from 3DMax models for data matching;

[0065] Preprocess the 3DMax model and instance object point cloud separately;

[0066] Assume that the front of the 3DMax model is aligned with the positive direction of the X-axis, and the top of the model is aligned with the positive direction of the Z-axis, extract the intersection lines of adjacent faces, and use a directional bounding box;

[0067] The endpoints of the line segments are used as object-level key points, and the corresponding line segment lengths are calculated to determine the scaling of each object;

[0068] (3)

[0069] in, and are the two endpoints of the target data segment, and are the two endpoints of the source data segment, is the scaling factor, is the Euclidean distance between two points;

[0070] The direction vector of the line segment is used to enhance the matching accuracy, calculate the center of gravity of the source data and the target data, and determine the direction vector from the midpoint of each line segment to the center of gravity, which is called ;

[0071] According to the normal vector of the surface and The angle between them is used to adjust the normal vector of the face patch, so that the face normal vector faces the direction of the constraint.

[0072] The horizontal plane is called , the vertical plane is called ,Will and The direction of the cross product is defined as the direction of the line segment;

[0073] A global optimization strategy is adopted to constrain the matching process by globally minimizing the angle between the instance object point cloud and the corresponding face normal vector in the 3DMax model, so that the 3DMax model can be fitted to the instance object, achieving the position, direction and size fitting of indoor objects.

[0074] (4)

[0075] is the number of faces, The source data The normal vector of a face, It is the first The normal vector of a face.

[0076] Optionally, the structural relationship diagram of constructing the indoor three-dimensional structured model specifically includes:

[0077] The structural relationship diagram of the indoor 3D structured model decomposes the indoor scene into multiple interconnected indoor single house units. Each house unit is divided into two categories: fixed structure and moving objects.

[0078] Fixed structures include: multiple walls, ceilings, floors, doors and windows, which form the basic framework of the interior space. Doors and windows serve as bridges connecting multiple spaces, and their locations determine the connectivity between adjacent houses.

[0079] Moving objects include: tables, chairs, bookshelves, cabinets, sofa furniture;

[0080] In the structural connection graph, each scene element is expressed as a node, and the structural elements that share the boundary are connected by an edge to form a structural network;

[0081] In indoor environments, doors are key points for space transformation, and their location information constitutes the connectivity of adjacent houses;

[0082] By utilizing the parallel and reverse normal vector characteristics between walls, matching walls are searched within the set distance and angle thresholds, the relationship between adjacent houses is confirmed, and a topological map of the room is drawn.

[0083] The present invention provides an indoor scene understanding and structured modeling method based on multi-source data fusion, the modeling method includes: multi-source data fusion of joint key point detection and description; multi-granularity understanding of indoor scenes based on semantic enhancement and multi-task collaboration; fusion of data-driven and model-driven reconstruction of indoor full-element three-dimensional structured models. A set of robust and robust indoor automated modeling methods is established to provide important technical support for indoor application services.

[0084] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0086] Figure 1 A flowchart of an indoor scene understanding and structured modeling method based on multi-source data fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0088] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0089] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0090] The present invention proposes a method for understanding and reconstructing a structured model of a complex indoor scene based on fusion of point cloud and image, including:

[0091] Firstly, high-precision fusion of multi-source data of joint key point detection and description is achieved to realize high-precision fusion of mobile measurement point cloud and image multimodal data under a unified network framework, providing high-quality data guarantee for indoor scene understanding and indoor 3D modeling.

[0092] Secondly, based on the multi-granularity understanding of indoor scenes with semantic enhancement and multi-task collaboration, a single house segmentation method that integrates semantic constraints and multi-labeled graph cuts is used to achieve coarse-grained understanding of indoor scenes. Task collaboration, inductive transfer, shared representation, etc. are used to improve the generalization ability of the network, and multi-level and fine-grained indoor scene understanding (component-level instance segmentation) is achieved.

[0093] Finally, the data-driven and model-driven methods are integrated to reconstruct the three-dimensional structured model of all indoor elements. The volume model is reconstructed using regular lines and surfaces, that is, the global consistency correction of error lines and the reconstruction of indoor structured models are achieved based on graph optimization theory; the 3DMax model of the indoor object is automatically matched with the instance segmentation result, and the three-dimensional model reconstruction of all indoor elements is realized in combination with the reconstructed indoor structured model. The present invention establishes a robust and robust indoor automated modeling method. The reconstructed indoor three-dimensional model is not only used for three-dimensional visualization, but also has the characteristics of semantics, monomerization, structuring, and lightweight. It provides indoor spatiotemporal data support for real-scene three-dimensional construction, urban fine governance, smart city construction, service guarantee, etc., and has broad application value.

[0094] High-precision fusion of multi-source data for indoor scenes with joint key point detection and description, including:

[0095] The mobile measurement system continuously collects indoor two-dimensional images and three-dimensional laser point cloud data. However, there are great differences between images and point clouds in terms of expression, data structure, spatial reference, data quality, etc., which leads to the limitations of multi-source data fusion such as low key point detection accuracy, difficulty in characterizing features with the same name, and high mismatch rate, which cannot meet the needs of three-dimensional indoor modeling. Therefore, the present invention proposes a high-precision fusion method for multi-source data with joint key point detection and description, and overcomes key technologies such as local feature calculation based on ultra-wide receptive field mechanism, key point detection based on the significance of comprehensive coordinate space and feature space, and coarse-to-fine optimization that takes into account key point detection and description, which can simultaneously realize functions such as 2D image registration, 3D point cloud registration, and 2D image and 3D point cloud fusion.

[0096] (1) Local feature descriptor calculation based on ultra-wide receptive field mechanism

[0097] Due to the irregularity and sparsity of the three-dimensional point distribution, there are inherent information changes between the two-dimensional pixels and the local areas of the corresponding three-dimensional points. In order to solve the inherent information changes between the two, the present invention proposes a local feature descriptor calculation method based on an ultra-wide receptive field mechanism. For 2D images, the ultra-wide receptive field mechanism is realized by using a dilated convolution with a gradually increasing dilation rate, and finally a feature map and a score map consistent with the resolution of the original image are generated. For 3D point clouds, the same strategy is used to improve the core point convolution network to learn the features of each three-dimensional point, and finally output the high-dimensional feature vector and significance of each point. The high-dimensional feature vector after L2 regularization is directly compared between the point cloud and the image, and the cosine similarity is used to establish the corresponding relationship of the same name. In training, it is expected that the feature values ​​of the same-name pixels and point pairs are adjusted to be as similar as possible.

[0098] (2) Key point detection based on the saliency of coordinate space and feature space

[0099] The present invention uses the significance of coordinate space and feature space to select key points. First, the most significant feature dimension is selected, and then the local maximum of the feature dimension is determined in the spatial neighborhood R. One point / pixel To become a key point, it needs to meet the following two conditions: saliency in feature space and saliency in coordinate space. In order to take into account the saliency of coordinate and feature space, it is expected to maximize the product of the two saliencies in all feature dimensions.

[0100] (3) A coarse-to-fine optimization strategy that takes into account both key point detection and description

[0101] In order to achieve joint optimization of 2D and 3D key point detection and description in one forward propagation, a coarse-to-fine loss function that takes into account both key point detection and description is designed, as shown in formula (1):

[0102] (1)

[0103] in, Relatively coarse supervision of each descriptor, Focus on supervising descriptors that are prone to mismatching, is the scale factor.

[0104] In order to improve the convergence of the feature description learning process, circle loss is used as the feature descriptor loss function. Key points are still repeatable and highly distinguishable under changes in viewpoints, light, etc. Therefore, the loss function is designed to encourage points with high distinguishability to have higher significance scores. The high-precision cross-modal data fusion method of joint key point detection and description realizes the fusion of multimodal image and point cloud data under a unified network framework, forming an enhanced point cloud with strong reality, high integrity, accurate position, and rich texture information, providing high-quality data guarantee for indoor scene understanding and three-dimensional model reconstruction.

[0105] Multi-granularity understanding of indoor scenes based on semantic enhancement and multi-task collaboration, including:

[0106] In view of the problems of complex spatial relationships, insufficient feature representation capabilities, poor generalization ability of independent task learning, and low segmentation accuracy of single data source in indoor scene understanding, the present invention proposes a key technology of multi-level and multi-granularity understanding of indoor scenes based on fused multi-source point cloud data with semantic enhancement and multi-task collaboration. First, in the problem of over-segmentation of multi-house and complexly connected indoor spaces, a single-house segmentation method integrating semantic constraints and multi-labeled graph cuts is proposed to segment the unordered point cloud into a single room with semantic information, so as to achieve coarse-grained understanding of indoor scenes. Secondly, for the component-level objects of a single house after segmentation, the present invention uses task collaboration, inductive transfer, shared representation, etc. based on the fused multi-source data to improve the generalization ability of the network, and achieve multi-level (i.e., positioning, semantics, shape) and fine-grained (i.e., component-level instance segmentation) indoor scene understanding.

[0107] (1) Single house segmentation method integrating semantic constraints and multi-labeled graph cuts

[0108] The present invention proposes a method for single house segmentation by integrating semantic constraints and multi-label graph cut model. First, the visibility analysis of the sampled moving trajectory points is performed. Then, the range of the visible point cloud is limited based on the location of the door, and the moving trajectory points are segmented into subspaces to form the initial division of the space. Finally, a multi-label graph cut energy function is constructed based on the similarity of the visible point clouds of adjacent trajectory points and the spatial smoothness, and similar visible point clouds are automatically clustered to complete the segmentation of a single house.

[0109] In single house segmentation, a multi-label graph cut energy function is constructed and minimized, as shown in (2), which includes the data cost and the smoothness cost.

[0110] (2)

[0111] Data items: ,in It is a trajectory point Assigned track segment The penalty value is With track segment The greater the visible point cloud coverage, the penalty value of the energy function Therefore, trajectory points tend to cluster into trajectory segments with greater similarity.

[0112] Smoothing term: , where the component It is used to regularize labels, that is, to penalize adjacent trajectory points for being assigned different label values. If adjacent trajectory points belong to the same space, the smoothing term is 0; if adjacent trajectory points do not belong to the same space, the greater the coverage of two adjacent trajectory points and the smaller the point spacing, the greater the penalty value of the smoothing term, which is close to 1. The smoothing term reduces the redundant subspace, so that adjacent trajectory points are clustered into the same house.

[0113] (2) Multi-level and fine-grained understanding of indoor scenes combined with multi-task deep learning

[0114] For each house after segmentation, a backbone feature learning network based on graph neural network and structure-attention mechanism is developed to analyze the neighborhood relationship composition and structural information in three-dimensional space and improve the feature learning representation ability. On the basis of the feature learning backbone network, a multi-task deep learning network integrating three-dimensional object detection, semantic segmentation and instance segmentation is developed. Multiple tasks promote each other, improve the generalization ability of the network, and realize multi-level (i.e., positioning, semantics, shape) and fine-grained (i.e., component-level instance segmentation) indoor scene understanding under a unified network framework.

[0115] 1) Backbone feature learning network based on graph neural network and structure-attention mechanism

[0116] First, for the original large-scale point set, a combination of traditional random sampling, spherical sampling, and regional sampling with improved autoencoders is used to extract a subset of the scene and reduce the number of elements in the point set. Then, a graph convolutional network is used to connect the relationships between points in the global or local range for the streamlined subset. Finally, for the primary feature expression completed by graph convolution processing, a deep feature extraction skeleton connected by a layered convolutional neural network can be constructed, and the attention mechanism is introduced in the dimension of the correlation of the structure in space to form a three-dimensional object cognition process that conforms to the human brain's cognition, improve cognitive ability layer by layer, complete the feature extraction of the entire scene, and form the construction of inter-class relationships within the scene.

[0117] 2) Object detection branch based on convolutional attention mechanism and non-local feature pyramid

[0118] The backbone network is a preliminary feature extraction, and further learning is needed to learn features that are more suitable for target detection and tracking tasks. Integrate spatial attention and channel attention into the implementation of point cloud convolution to learn feature information that is more beneficial to 3D target detection. Use non-local feature pyramids to fuse information based on the global similarity of features to achieve deformable feature extraction of 3D targets of different shapes and scales, thereby achieving more accurate 3D bounding box prediction. Construct a Gaussian distribution heat map of irregular shape point clouds based on the center point of the 3D bounding box.

[0119] 3) Semantic segmentation branch based on structural primitive relationship combination reasoning

[0120] The accuracy of semantic segmentation depends greatly on the combination relationship between spatial structural primitives and the retention of point-by-point positioning information. Therefore, the semantic segmentation method based on structural primitive relationship combination reasoning of the present invention mainly includes key steps such as spatial structural primitive expression, relationship reasoning and combination generalization and point-by-point spatial information recovery.

[0121] 4) Instance segmentation branch considering semantic boundary information and target location information

[0122] Instance segmentation requires the effective combination of semantic information, spatial boundary information and positioning information to obtain objectified information with spatial and semantic consistency. The present invention utilizes the positioning information of the target detection branch method and the category information of the semantic segmentation branch, and on this basis transforms the instance segmentation problem into two coupled processes: confidence matrix prediction and boundary positioning mask prediction.

[0123] Integrate data-driven and model-driven methods to reconstruct indoor structured models, including:

[0124] In view of the complex object structure, diverse morphology, and data missing characteristics of coarse-grained segmentation (single house) and fine-grained segmentation (instance objects in a single house) of indoor scenes, it is difficult to reconstruct indoor structured models with high precision and automatically by relying on a single modeling strategy. The present invention integrates data-driven and model-driven modeling strategies to reconstruct indoor three-dimensional models. For the coarse-grained segmentation results of indoor scenes (single house indoors), the indoor house model is reconstructed based on data-driven mixed constraints of lines and surfaces. For the fine-grained segmentation results (furniture such as tables, chairs, bookshelves, cabinets, sofas, etc.), the model is driven to automatically match the 3DMax model with the point cloud of instance segmentation to achieve model fitting of indoor objects. The indoor object model and the house model are combined to achieve the reconstruction of the indoor full-element three-dimensional model, providing high-quality data guarantee for indoor location services.

[0125] (1) Data-driven reconstruction of indoor house models based on mixed constraints of lines and surfaces

[0126] For the segmented single house, horizontal slices are taken at a certain height from the ceiling, and the point cloud of the horizontal slices is converted into a binary image. The line segment detector (LSD) is used to extract straight lines from the binary image. However, due to the problems of information loss, voids, noise, etc. of the laser point cloud, the extraction results mainly have angle deviation, distance deviation, excessive redundancy, and incomplete boundaries. The present invention performs global consistency correction on the angle and distance deviation of the line segment based on the image polygon block graph optimization theory and the G2o solver. After the global correction of the line segment error term, the indoor scene is still expressed by independent small line segments with different labels, and the redundancy is too large. The region growing algorithm is used to cluster line segments with similar features to determine the main line segment.

[0127] Due to the partial missing of LiDAR point cloud, the extracted structural lines are incomplete. In order to reduce the impact of line segment missing on model reconstruction, the geometric structure information of the main line segment is combined with the semantic information of the segmented single house to complete the structural lines of the model and reconstruct the indoor three-dimensional model with correct topological relationship. First, the main line segment is extended to form a closed two-dimensional plane. The lines in the plane have a topological structure. The segmented single house point cloud is projected onto a two-dimensional polygonal plane, and each grid is assigned a label according to the projected point cloud. , including outdoor tags ; Each main line segment is divided into a one-dimensional uniform grid, and the one-dimensional grid label vector is determined by the projected small line segment. The label set is , the labeled line segments are used to separate the grids belonging to different houses. The energy function is minimized by multi-labeled graph cuts, and the two-dimensional polygons and line segments are globally optimized to construct the plane model. Then, the three-dimensional structured model is reconstructed according to the elevation information of the ceiling and floor planes. This method can effectively preserve the detailed features of the indoor scene.

[0128] (2) Model-driven fitting of indoor instance segmentation object models

[0129] For indoor instance segmentation objects (furniture such as tables, chairs, bookshelves, cabinets, sofas, etc.), 3DMax is used to build a template library, and an automatic registration method between instance segmentation point cloud and template library model is proposed to achieve consistent fitting of the position, direction and size of indoor objects, combine indoor objects and room models together, and generate a full-element 3D model of the interior.

[0130] The 3DMax model is regarded as the source data, and the instance segmentation point cloud is used as the target data. The source data and the target data have the same semantic labels, but there are certain differences in the size range. Object-level key points are defined as the two endpoints of the intersection line of adjacent faces, which have extremely high robustness and accuracy and are suitable for indoor scenes with severe occlusion. Object-level key points are extracted from the 3DMax model for data matching. First, the 3DMax model and the instance object point cloud are preprocessed separately. It is assumed that the front of the 3DMax model is aligned with the positive direction of the X-axis, and the top of the model is aligned with the positive direction of the Z-axis. On this basis, the intersection lines of adjacent facets are extracted and oriented bounding boxes are used. The endpoints of these line segments are used as object-level key points. In order to determine the scaling of each object, the corresponding line segment length needs to be calculated.

[0131] (3)

[0132] in, and are the two endpoints of the target data segment, and are the two endpoints of the source data segment, is the scaling factor, is the Euclidean distance between two points.

[0133] Since the matches between key points are non-unique, there are potential matching pairs.

[0134] The present invention enhances the matching accuracy by introducing the direction vector of the line segment, including:

[0135] First, the centroids of the source data (3DMax model) and the target data (instance segmentation point cloud) are calculated. Then, the direction vector from the midpoint of each line segment to the centroid is determined, which is called To enforce the orientation constraint, the face normal vector is The angle between the plane and the surface is used to adjust the normal vector of the face. This adjustment makes the surface normal vector point in the direction of the constraint. The horizontal plane is called , the vertical plane is called ,Will and The cross product direction of is defined as the direction of the line segment. This method makes the two pairs of object-level key points ambiguous, and considers introducing the centroid point. When the indoor object is partially occluded, the centroid will be unstable.

[0136] In order to overcome the problem, the present invention adopts a global optimization strategy to constrain the matching process by globally minimizing the angle between the instance object point cloud and the corresponding face normal vector in the 3DMax model, so that the 3DMax model can be accurately fitted to the instance object and the position, direction and size fitting of indoor objects can be achieved.

[0137] (4)

[0138] in, is the number of faces, The source data The normal vector of a face, It is the first The normal vector of a face.

[0139] (3) Structural relationship diagram of the indoor three-dimensional structural model

[0140] The structural relationship diagram of the indoor three-dimensional structured model is the basis for realizing indoor positioning services. It not only has the geometric form of the indoor space, but also contains the logical and physical relationships between spatial elements. The structural relationship diagram of the indoor three-dimensional structured model decomposes the indoor scene into multiple interconnected indoor single-room units, and each room unit is divided into two categories: fixed structure and mobile object. Fixed structures include: multiple walls, ceilings, floors, doors and windows, etc., which constitute the basic framework of the indoor space. Among them, doors and windows are bridges connecting different spaces, and their positions determine the connectivity between adjacent houses. Mobile objects include: furniture such as tables, chairs, bookshelves, cabinets, sofas, etc. In the structural connection diagram, each scene element is expressed as a node, and the structural elements with shared boundaries are connected by an edge to form a structural network. In the indoor environment, the door is the key point of space conversion, and its location information constitutes the connectivity of adjacent houses. In addition, by using the parallel and reverse normal vector characteristics between walls, matching walls are searched within the set distance and angle thresholds to confirm the relationship between adjacent houses, so as to draw the topological map of the room. When a home space is connected to three or more other homes and is equipped with multiple doors, the space is likely to be identified as a corridor, and the indoor positioning service provides richer and more accurate contextual information.

[0141] Beneficial effects: The reconstructed indoor 3D model can not only be used for 3D visualization, but also has the characteristics of semantics, monomerization, structuring and lightweight. It provides indoor spatiotemporal data support for the capital's real-scene 3D construction, urban fine governance, smart city construction, service guarantee, etc., and has broad application value.

[0142] The present invention establishes a set of stable and robust indoor automatic modeling methods, providing important technical support for indoor application services.

[0143] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for indoor scene understanding and structured modeling based on multi-source data fusion, characterized in that: The modeling method comprises: Multi-source data fusion of indoor scenes with joint key point detection and description; Multi-granularity understanding of indoor scenes based on semantic enhancement and multi-task collaboration, including: In order to solve the problem of over-segmentation of indoor spaces with multiple houses and complex connections, a single-house segmentation method integrating semantic constraints and multi-labeled graph cuts is proposed to segment the unordered point cloud into single rooms with semantic information, thus achieving coarse-grained understanding of indoor scenes. According to the component-level objects of a single house after segmentation, based on the fused multi-source data, the network generalization ability is improved by using task collaboration, inductive transfer, and shared representation to achieve multi-level and fine-grained indoor scene understanding, where the multi-level includes: positioning, semantics, and shape; the fine-grained includes component-level instance segmentation; The multi-level and fine-grained indoor scene understanding specifically includes: According to each house after segmentation, a backbone feature learning network based on graph neural network and structure-attention mechanism is developed to analyze the neighborhood relationship composition and structural information in three-dimensional space; based on the feature learning backbone network, a multi-task deep learning network integrating three-dimensional object detection, semantic segmentation, and instance segmentation is developed to achieve multi-level and fine-grained indoor scene understanding under a unified network framework; The backbone feature learning network based on graph neural network and structure-attention mechanism includes: First, for the original large-scale point set, we use a combination of traditional random sampling, spherical sampling, regional sampling and improved autoencoders to extract a subset of the scene and reduce the number of elements in the point set. Then, a graph convolutional network is used to connect the relationships between points in the global or local range on the streamlined subset; Finally, for the primary feature expression completed by graph convolution processing, a deep feature extraction skeleton connected by a hierarchical convolutional neural network is constructed, and the attention mechanism is introduced in the dimension of the correlation of the structure in space to form a three-dimensional object cognition process that conforms to the human brain's cognition, improve the cognitive ability layer by layer, complete the feature extraction of the entire scene, and form the construction of the relationship between classes within the scene; The object detection branch based on convolutional attention mechanism and non-local feature pyramid includes: Integrate spatial attention and channel attention into the implementation of point cloud convolution to learn feature information that is more beneficial to 3D object detection; Use non-local feature pyramids to fuse information based on the global similarity of features, extract deformable features of 3D objects of different shapes and scales, and predict 3D bounding boxes; construct a Gaussian distribution heat map of irregular shape point clouds based on the center point of the 3D bounding box; The semantic segmentation branch based on structural primitive relationship combination reasoning includes: The accuracy of semantic segmentation depends greatly on the combination relationship between spatial structural primitives and the retention of point-by-point positioning information. Semantic segmentation methods include: spatial structural primitive expression, relationship reasoning and combination generalization, and point-by-point spatial information recovery; The instance segmentation branch that takes into account semantic boundary information and target location information includes: By using the positioning information of the target detection branch method and the category information of the semantic segmentation branch, the instance segmentation problem is transformed into two coupled processes: confidence matrix prediction and boundary positioning mask prediction; Integrate data-driven and model-driven approaches to reconstruct a three-dimensional structured model of all indoor elements.

2. The method for indoor scene understanding and structured modeling based on multi-source data fusion according to claim 1, characterized in that: The indoor scene multi-source data fusion of joint key point detection and description specifically includes: (1) Calculation of local feature descriptors based on the ultra-wide receptive field mechanism, including: for 2D images, using dilated convolution with a gradually increasing dilation rate to achieve an ultra-wide receptive field mechanism, and finally generating feature maps and score maps with the same resolution as the original image; For 3D point clouds, the same strategy is used to improve the core point convolutional network to learn the features of each 3D point, and finally output the high-dimensional feature vector and saliency of each point; The high-dimensional feature vector after L2 regularization is directly compared between the point cloud and the image, and the cosine similarity is used to establish the corresponding relationship of the same name; During training, it is expected that the feature values ​​of pixels and point pairs with the same name will be adjusted to be as similar as possible; (2) Key point detection based on the saliency of the coordinate space and feature space, including: selecting the most significant feature dimension and determining the local maximum of the feature dimension in the spatial neighborhood R; One dot / pixel To become a key point, two conditions must be met: saliency in feature space and saliency in coordinate space. In order to take into account the saliency of both coordinate and feature space, the product of the two saliencies must be maximized. (3) A coarse-to-fine optimization strategy that takes into account both key point detection and description, including: In one forward propagation, the detection and description of 2D and 3D key points are jointly optimized, and a coarse-to-fine loss function that takes into account both key point detection and description is designed, as shown in formula (1): （1） in, Relatively coarse supervision of each descriptor, Focus on supervising descriptors that are prone to mismatching, is the scale factor; Using circle loss as the feature descriptor loss function to improve the convergence of the feature description learning process; The key points are still repeatable and highly distinguishable when the viewpoint and light change. The loss function is designed to encourage points with high distinguishability to have significant scores. The high-precision cross-modal data fusion method of joint key point detection and description realizes the fusion of multimodal image and point cloud data under a unified network framework to form an enhanced point cloud, providing data guarantee for indoor scene understanding and three-dimensional model reconstruction.

3. The method for indoor scene understanding and structured modeling based on multi-source data fusion according to claim 1, characterized in that: The single house segmentation method integrating semantic constraints and multi-labeled graph cuts specifically includes: Perform visual analysis on sampled moving trajectory points, including: The visible point cloud range is limited based on the position of the door, and the moving trajectory points are segmented into subspaces to form the initial division of the space; Based on the visual point cloud similarity and spatial smoothness of adjacent trajectory points, a multi-labeled graph cut energy function is constructed to minimize it, and similar visual point clouds are automatically clustered to complete the segmentation of a single house. （2） Data items: ,in, It is a trajectory point Assigned track segment The penalty value is With track segment The greater the visible point cloud coverage, the penalty value of the energy function The smaller it is, the trajectory points tend to cluster in trajectory segments with greater similarity; It is the weight factor of the data item, which controls the weight of the spatial similarity influencing factor; Smoothing term: , where the component It is used to regularize labels and penalize adjacent trajectory points that are assigned different label values. If adjacent trajectory points belong to the same space, the smoothing term is 0. If adjacent trajectory points do not belong to the same space, the greater the coverage of two adjacent trajectory points and the smaller the point spacing, the greater the penalty value of the smoothing term, which is close to 1. The smoothing term reduces the redundant subspace, so that adjacent trajectory points are clustered into the same house. is the weight of the factor that controls the smoothness of the space, The larger the value, the smoother the paths between adjacent track points.

4. The method for indoor scene understanding and structured modeling based on multi-source data fusion according to claim 1, characterized in that: The fusion of data-driven and model-driven reconstruction of the indoor full-element three-dimensional structured model specifically includes: Data-driven reconstruction of indoor house models based on line and surface mixed constraints; Fitting indoor instance segmentation object model based on model-driven; Construct a structural relationship diagram of the indoor three-dimensional structured model.

5. The method for indoor scene understanding and structured modeling based on multi-source data fusion according to claim 4 is characterized in that: The data-driven reconstruction of the indoor house model based on line and surface mixed constraints specifically includes: According to the segmented single house, horizontal slices are made at a certain height from the ceiling, the point cloud of the horizontal slices is converted into a binary image, and a line segment detector is used to extract straight lines from the binary image; Based on the image polygon block optimization theory and G2o solver, the angle and distance deviation of the line segment are corrected globally. After the global correction of the line segment error term, the indoor scene is still expressed by independent small line segments with different labels, and the redundancy is too large. The region growing algorithm is used to cluster the line segments with similar features and determine the main line segments; Combine the geometric structure information of the main line segment with the semantic information of the segmented single house to complete the structural line of the model and reconstruct the indoor 3D model with correct topological relationship; Extending the main line segments forms a closed two-dimensional plane graph, and the lines in the plane graph have a topological structure; Project the segmented single house point cloud onto a two-dimensional polygonal plane, and assign labels to each grid based on the projected point cloud. , including outdoor tags ; Each main line segment is divided into a one-dimensional uniform grid. The one-dimensional grid label vector is determined by the projected small line segment. The label set is ,Line segments with labels are used to separate grids belonging to different houses; The energy function is minimized by multi-labeled graph cuts, and the two-dimensional polygons and line segments are globally optimized to construct the plane model. The three-dimensional structured model is reconstructed according to the elevation information of the ceiling and floor planes.

6. The method for indoor scene understanding and structured modeling based on multi-source data fusion according to claim 4, characterized in that: The model-driven fitting of the indoor instance segmentation object model specifically includes: For indoor instance segmentation objects, 3DMax is used to build a template library, and an automatic registration method between instance segmentation point cloud and template library model is proposed to combine indoor objects and room models to generate a full-factor indoor 3D model. The indoor full-element three-dimensional model is used as source data, and the instance segmentation point cloud is used as target data; Object-level key points are defined as the two endpoints of the intersection line of adjacent faces; Extract object-level key points from 3DMax models for data matching; Preprocess the 3DMax model and instance object point cloud separately; Assume that the front of the 3DMax model is aligned with the positive direction of the X-axis, and the top of the model is aligned with the positive direction of the Z-axis, extract the intersection lines of adjacent faces, and use a directional bounding box; The endpoints of the line segments are used as object-level key points, and the corresponding line segment lengths are calculated to determine the scaling of each object; （3） in, and are the two endpoints of the target data segment, and are the two endpoints of the source data segment, is the scaling factor, is the Euclidean distance between two points; The direction vector of the line segment is used to enhance the matching accuracy, calculate the center of gravity of the source data and the target data, and determine the direction vector from the midpoint of each line segment to the center of gravity, which is called ; According to the normal vector of the surface and The angle between them is used to adjust the normal vector of the face patch, so that the face normal vector faces the direction of the constraint. The horizontal plane is called , the vertical plane is called ,Will and The direction of the cross product is defined as the direction of the line segment; A global optimization strategy is adopted to constrain the matching process by globally minimizing the angle between the instance object point cloud and the corresponding face normal vector in the 3DMax model, so that the 3DMax model can be fitted to the instance object, achieving the position, direction and size fitting of indoor objects. （4） is the number of faces, The source data The normal vector of a face, It is the first The normal vector of a face.

7. The method for indoor scene understanding and structured modeling based on multi-source data fusion according to claim 4, characterized in that: The structural relationship diagram of constructing the indoor three-dimensional structured model specifically includes: The structural relationship diagram of the indoor 3D structured model decomposes the indoor scene into multiple interconnected indoor single house units. Each house unit is divided into two categories: fixed structure and moving objects. Fixed structures include: multiple walls, ceilings, floors, doors and windows, which form the basic framework of the interior space. Doors and windows serve as bridges connecting multiple spaces, and their locations determine the connectivity between adjacent houses. Moving objects include: tables, chairs, bookshelves, cabinets, sofa furniture; In the structural connection graph, each scene element is expressed as a node, and the structural elements that share the boundary are connected by an edge to form a structural network; In indoor environments, doors are key points for space transformation, and their location information constitutes the connectivity of adjacent houses; By utilizing the parallel and reverse normal vector characteristics between walls, matching walls are searched within the set distance and angle thresholds, the relationship between adjacent houses is confirmed, and a topological map of the room is drawn.

Images