Indoor 3D scene understanding method and terminal based on multimodal feature embedding

Abstract

The present invention discloses a method and a terminal for understanding indoor three-dimensional scenes with multimodal feature embedding, and the method comprises: obtaining a two-dimensional image and preprocessed point cloud data, and performing multimodal feature extraction on the two-dimensional image and the preprocessed point cloud data; fusing the extracted multimodal features to obtain multimodal fusion features, and optimizing a multimodal feature embedded indoor three-dimensional scene understanding model through the multimodal fusion features; extracting semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and outputting a semantic classification optimization result; the present invention proposes a new method for understanding indoor three-dimensional scenes by combining a Transformer-based 3D semantic instance segmentation model with multimodal features, so as to improve the accuracy and comprehensiveness of indoor three-dimensional scene understanding.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor three-dimensional scene understanding, and in particular to a method and a terminal for indoor three-dimensional scene understanding with multimodal feature embedding. Background Art

[0002] At present, multimodal feature embedding has attracted widespread attention in the field of indoor 3D scene understanding research, providing key technical support for achieving more comprehensive and profound scene understanding, and also reflecting the huge potential of multimodal feature fusion. First, researchers focus on how to fuse information from multiple sensors, including RGB cameras, depth sensors, and lidar. These sensors provide rich scene feature information such as color, depth, distance, and texture. Multimodal fusion helps the system perceive and understand the indoor environment more comprehensively. Secondly, semantic segmentation and object detection are key tasks in this field, which are used to divide scenes into different objects and regions and detect objects in them.

[0003] Indoor 3D scene understanding technology has not only attracted widespread interest in academia, but also played an important role in many practical application fields. It is applied to indoor navigation, smart home, security monitoring and other fields. For example, it can be used to build smart home systems, monitor abnormal conditions in indoor environments, or assist robots in navigating indoor environments.

[0004] In general, the research field of indoor 3D scene understanding with multimodal feature embedding is constantly evolving and will continue to make breakthroughs in the future. Multimodal data fusion enables computers to make more full use of scene information, thereby understanding the indoor environment more comprehensively and accurately, providing strong support for intelligent applications and services.

[0005] With the increasing complexity of indoor space structures and the increasing indoorization of human activities, the demand for indoor space research is growing. Correspondingly, with the rapid development of data acquisition equipment such as laser scanning and depth sensors, the acquisition of indoor three-dimensional point cloud data has become more convenient and accurate, and high-precision indoor scene data of different modalities has become more abundant. However, traditional methods rely on a single sensor or limited data modality, and cannot fully utilize scene information, which limits the comprehensiveness and accuracy of scene understanding.

[0006] Therefore, the prior art needs to be improved. Summary of the invention

[0007] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a method and terminal for indoor three-dimensional scene understanding with multimodal feature embedding, so as to solve the technical problem that the traditional indoor three-dimensional scene understanding technology has low utilization rate of scene information.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] In a first aspect, the present invention provides a method for understanding an indoor three-dimensional scene by embedding multimodal features, comprising:

[0010] Acquire a two-dimensional image and preprocessed point cloud data, and perform multimodal feature extraction on the two-dimensional image and the preprocessed point cloud data;

[0011] The extracted multimodal features are fused to obtain multimodal fusion features, and the multimodal feature embedded indoor three-dimensional scene understanding model is optimized by the multimodal fusion features;

[0012] Semantic features are extracted from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and an optimized instance mask and optimized semantic information are output.

[0013] In one implementation, the acquiring of the two-dimensional image and the preprocessed point cloud data, and performing multimodal feature extraction on the two-dimensional image and the preprocessed point cloud data, includes:

[0014] The two-dimensional image is acquired, a mask corresponding to a single target is obtained through target recognition and extraction, and visual feature extraction is performed through a language-image comparison pre-training model to obtain a two-dimensional single target global feature;

[0015] The preprocessed point cloud data is obtained, multimodal features are extracted from the preprocessed point cloud data, and instance segmentation is performed through a 3D semantic instance segmentation model based on deep learning to obtain semantic global features, single instance global features, and multi-view fusion global features.

[0016] In one implementation, the step of obtaining the preprocessed point cloud data includes:

[0017] Acquire point cloud data, perform quantization, densification, smoothing, compression and supervoxelization operations on the point cloud data, and obtain the preprocessed point cloud data.

[0018] In one implementation, the preprocessed point cloud data is subjected to multimodal feature extraction, and instance segmentation is performed using a 3D semantic instance segmentation model based on deep learning to obtain semantic global features, single instance global features, and multi-view fusion global features, including:

[0019] Performing semantic segmentation on the preprocessed point cloud data and generating corresponding sample labels, extracting corresponding type semantics according to the sample labels, and performing semantic feature encoding on the type semantics to obtain the semantic-based global features;

[0020] Performing instance segmentation on the preprocessed point cloud data by using the deep learning-based 3D semantic instance segmentation model, extracting a single instance, and obtaining the single instance global feature by encoding the instance feature;

[0021] According to the instance result obtained by segmentation of the 3D semantic instance segmentation model based on deep learning, a corresponding instance mask and corresponding semantic information are extracted, and a multi-view depth map is projected according to the instance mask and the corresponding semantic information to obtain a multi-view depth map;

[0022] Multi-view feature extraction is performed through a language-image comparison pre-training model, and the multi-view features are aggregated, and the multi-view fusion-based global feature is obtained through fusion feature encoding.

[0023] In one implementation, fusing the extracted multimodal features to obtain multimodal fusion features, and optimizing the multimodal feature embedded indoor three-dimensional scene understanding model through the multimodal fusion features, includes:

[0024] The two-dimensional single target global feature, the semantic-based global feature, the single instance global feature and the multi-view fusion-based global feature are fused to obtain the multimodal fusion feature;

[0025] The multimodal feature embedded indoor three-dimensional scene understanding model is retrained and adjusted through the multimodal fusion feature to optimize the model parameter configuration.

[0026] In one implementation, fusing the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature to obtain the multimodal fusion feature includes:

[0027] Performing text generation and text alignment operations on the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature to obtain aligned multimodal features;

[0028] The aligned multimodal features are fused to obtain the multimodal fusion features.

[0029] In one implementation, the operations of performing text generation and text alignment on the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature to obtain the aligned multimodal feature include:

[0030] Through a text generation model, the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature are mapped to a natural language description space to obtain a corresponding text description sequence;

[0031] A text alignment operation is performed on the text description sequence, and the result of the text alignment is analyzed and verified to obtain the aligned multimodal features.

[0032] In one implementation, the extracting semantic features of the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and outputting the optimized instance mask and the optimized semantic information, includes:

[0033] Extracting semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model to obtain the optimized instance mask and the optimized semantic information corresponding to the preprocessed point cloud data;

[0034] The optimized instance mask and the optimized semantic information are output.

[0035] In a second aspect, the present invention also provides a terminal, comprising: a processor and a memory, wherein the memory stores a multimodal feature embedded indoor three-dimensional scene understanding program, and when the multimodal feature embedded indoor three-dimensional scene understanding program is executed by the processor, it is used to implement the operation of the multimodal feature embedded indoor three-dimensional scene understanding method as described in the first aspect.

[0036] In a third aspect, the present invention also provides a computer-readable storage medium, which stores a multimodal feature embedded indoor three-dimensional scene understanding program, and when the multimodal feature embedded indoor three-dimensional scene understanding program is executed by a processor, it is used to implement the operation of the multimodal feature embedded indoor three-dimensional scene understanding method as described in the first aspect.

[0037] The present invention adopts the above technical solution to achieve the following effects:

[0038] The present invention obtains two-dimensional images and pre-processed point cloud data, and extracts multimodal features of two-dimensional images, pre-processed point cloud data features, projection image fusion features and text semantic features; the extracted multimodal features are fused to obtain multimodal fusion features, and the multimodal feature embedded indoor three-dimensional scene understanding model is trained through the multimodal fusion features, and the model parameters are optimized; the semantic features of the pre-processed point cloud data are extracted according to the multimodal feature embedded indoor three-dimensional scene understanding model after training optimization, and more accurate semantic classification information and instance masks are output, so as to realize the multimodality of feature level. The multimodal feature data are fused by deep learning technology, and the model parameters are trained and optimized to realize the refined understanding of indoor three-dimensional scenes, so that the model's understanding of indoor three-dimensional scenes is more accurate and comprehensive, and the problem of single use mode of scene information data and low utilization rate of scene information in traditional technology is solved. More possibilities are provided for the intelligent application and management of indoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of a method for indoor three-dimensional scene understanding by embedding multimodal features in one implementation of the present invention.

[0041] Figure 2 It is a concept and flow chart of a Transformer-based 3D semantic instance segmentation model in one implementation of the present invention.

[0042] Figure 3 It is a framework diagram of an indoor three-dimensional scene understanding method with multimodal feature embedding in one implementation of the present invention.

[0043] Figure 4 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0044] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] The present invention provides a method, terminal and storage medium for indoor three-dimensional scene understanding with multimodal feature embedding. In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] Exemplary Methods

[0047] With the increasing complexity of indoor space structures and the increasing indoorization of human activities, the demand for indoor space research is growing. Correspondingly, with the rapid development of data acquisition equipment such as laser scanning and depth sensors, the acquisition of indoor 3D point cloud data has become more convenient and accurate, and high-precision indoor scene data of different modalities has become more abundant. However, traditional methods rely on a single sensor or limited data modality and cannot fully utilize scene information, which limits the comprehensiveness and accuracy of scene understanding. Therefore, a method for indoor 3D scene understanding that integrates multimodal data is needed to overcome these limitations.

[0048] In response to the above technical problems, an embodiment of the present invention provides a method for indoor three-dimensional scene understanding with multimodal feature embedding, with the aim of proposing a new method for indoor three-dimensional scene understanding based on a Transformer (deep learning) 3D semantic instance segmentation model combined with multimodal features, so as to improve the accuracy and comprehensiveness of indoor three-dimensional scene understanding.

[0049] like Figure 1 As shown, an embodiment of the present invention provides a method for understanding an indoor three-dimensional scene by embedding multimodal features, comprising the following steps:

[0050] Step S100, acquiring a two-dimensional image and pre-processed point cloud data, and performing multimodal feature extraction on the two-dimensional image and the pre-processed point cloud data.

[0051] In this embodiment, the multimodal feature embedded indoor three-dimensional scene understanding method is applied to a terminal, and the terminal includes but is not limited to: computers and mobile terminals and other devices; the terminal is provided with a training migration platform for the multimodal feature embedded indoor three-dimensional scene understanding model.

[0052] In this embodiment, a method for understanding indoor three-dimensional scenes by combining a Transformer-based 3D semantic instance segmentation model with multimodal features is proposed, aiming to improve the model's ability to understand indoor three-dimensional scenes, enable it to more accurately identify and classify various elements in the scene, improve the accuracy and comprehensiveness of indoor three-dimensional scene understanding, and provide more possibilities for intelligent application and management of indoor scenes; in this embodiment, the proposed method for understanding indoor three-dimensional scenes by embedding multimodal features utilizes multimodal data for multimodal feature extraction, and fuses multimodal feature information to realize automated modeling and understanding of indoor environments, including laser scanning point clouds, multi-view projection images based on point clouds, two-dimensional images, semantic label information and other multimodal data.

[0053] The technical solution of this embodiment mainly includes two parts: Multimodal data fusion: This embodiment will integrate multimodal features based on point cloud and its variant data, multi-view feature fusion information, multi-resolution feature information of 3D point cloud, two-dimensional image feature information and semantic information, etc., to achieve comprehensive indoor scene information collection. Deep learning technology: Based on the Transformer-based 3D semantic instance segmentation model (Mask3D model), feature extraction, semantic information optimization and feature fusion technologies of CLIP network (a multimodal learning model) are embedded to achieve semantic analysis and comprehensive understanding of indoor three-dimensional scenes.

[0054] In this embodiment, based on point cloud data, relevant preprocessing operations are performed on the point cloud data to extract 3D point cloud multi-resolution feature information, 2D image features, depth information, semantic information, etc., to achieve multimodality of feature levels, and these multimodal feature information are fused to achieve a comprehensive understanding of indoor three-dimensional scenes using deep learning technology.

[0055] Specifically, in an implementation of this embodiment, step S100 includes the following steps:

[0056] Step S110, acquiring the two-dimensional image, obtaining a mask corresponding to a single target through target recognition and extraction, and extracting visual features through a language-image comparison pre-training model to obtain a two-dimensional single target global feature.

[0057] In this embodiment, combined with the two-dimensional image data of indoor scenes, the mask corresponding to the single instance is obtained through target recognition and extraction, and the visual feature extraction is performed through the language-image contrast pre-training model (CLIP model) to obtain the global features of the two-dimensional single instance, which provides an important data basis for subsequent indoor scene understanding.

[0058] Step S120, obtaining the preprocessed point cloud data, performing multimodal feature extraction on the preprocessed point cloud data, and performing instance segmentation through a Transformer-based 3D semantic instance segmentation model to obtain semantic global features, single instance global features, and multi-view fusion-based global features.

[0059] In this embodiment, Mask3D is an end-to-end 3D instance segmentation model based on Transformer. It is a technology for object detection and segmentation. It can well identify objects in indoor scenes and extract their features, which helps the model understand the semantic information and structure of the scene. Figure 2 As shown in Figure 1, the original Transformer-based 3D semantic instance segmentation model mainly consists of three modules: a feature extraction backbone network (Res16UNet34C), a Transformer decoding layer built by mask modules, and a query module based on the Transformer decoding layer. The core of the model is the instance query, each instance query should represent an object instance in the scene and predict the corresponding point-level instance mask.

[0060] In this embodiment, the indoor 3D scene instance information extraction based on feature optimization Mask3D includes the mask of each instance in the scene and its corresponding semantic information. The main steps include:

[0061] (1) Multi-scale features extracted based on feature backbone Instance queries are iteratively refined through the Transformer decoder, making instance queries cross-focus on multi-scale point features And self-focus on other instance queries, after several repeated iterations, a refined final set of instance queries is finally generated ;

[0062] (2) Mask Module uses instance query and point features , and returns a semantic class and a binary instance mask for each query based on the dot product between the point feature and the instance query. The instance information extracted by Mask3D can provide intermediate data support for advanced point cloud multi-view projection and feature extraction.

[0063] Specifically, in an implementation of this embodiment, step S120 includes the following steps:

[0064] Step S121, acquiring point cloud data, and performing quantization, densification, smoothing, compression and supervoxelization operations on the point cloud data to obtain the preprocessed point cloud data.

[0065] In this embodiment, point cloud data, as one of the most popular three-dimensional data nowadays, can provide three-dimensional information of indoor environments. Point cloud data collected in indoor environments usually include various objects and structures in the room, such as furniture, floors, walls, ceilings, and various objects placed in the room. However, the original point cloud data may be subject to some interference during collection, resulting in problems such as noise, deviation, and alignment and distortion removal. Therefore, for the subsequent use of point clouds and specific input needs, it is necessary to perform pre-processing operations on the point cloud data, including quantization, densification, smoothing, compression, and super-voxelization, and high-fidelity projection of multi-view depth images based on point clouds, so as to provide good data preparation for subsequent feature extraction of different modalities.

[0066] Specifically, the preprocessing operations include: quantization, densification, smoothing, compression and supervoxelization. The data preprocessing in this embodiment is divided into different stages, and the processing in each stage is different. For the original data, the main preprocessing operations are as follows:

[0067] Quantization: To project the M different views onto the image, create a 3D grid initialized to zero. ,in Representation Grid The elements in belong to the set of real numbers, , and represents its spatial resolution, Indicates height, Indicates width, represents the depth dimension along the viewing plane, Represented as a 3D grid containing sparse depth values.

[0068] Densification: In order to solve the problem that some voxels are empty due to the sparse point cloud, the density of the grid is increased through local minimum pooling operation to ensure visual continuity. The specific method is to select the minimum voxel value in the local space window and Each voxel in the image can be redistributed, which can effectively fill the originally blank voxels between sparse points with reasonable depth values, while the background voxels remain empty, thereby deriving a dense and three-dimensional spatial shape representation.

[0069] Smoothing: In order to solve the problem of artifacts on some 3D surfaces caused by local pooling operations, a non-parametric Gaussian kernel is used for shape smoothing and noise filtering. With appropriate kernel size and variance, filtering can remove spatial noise caused by densification while maintaining the clarity of edges and corners in the original 3D shape, and finally a more compact and smooth shape represented by a 3D mesh can be obtained.

[0070] Compression: At the end of preprocessing, the The depth dimension of the projected depth map Specifically, by extracting the minimum value of each depth channel as the value of each pixel position, and repeating it three times as the RGB intensity, The depth channel of is min-pooled to simply implement grid-based projection.

[0071] Supervoxelization operation based on point cloud data: Point cloud data may be very dense. In order to extract useful information, the point cloud data is divided into larger parts for subsequent analysis by using supervoxelization operation. This helps to reduce computational complexity and extract higher-level features. Supervoxelization is a method of dividing a point cloud into supervoxels with semantic information, so that adjacent points are aggregated into larger spatial units. By supervoxelizing the original point cloud and performing feature extraction and fusion on these supervoxels, the expressive power of the original point cloud is enhanced, and the point cloud data can be "pre-segmented" to a great extent, so as to achieve the effect of limiting the range of subsequent feature extraction and instance, and avoid problems such as instance confusion across distances.

[0072] In this embodiment, the data quality and accuracy can be improved by preprocessing the point cloud data, providing a better basis for subsequent data analysis, processing and utilization.

[0073] Step S122, performing semantic segmentation on the preprocessed point cloud data and generating corresponding sample labels, extracting corresponding type semantics according to the sample labels, performing semantic feature encoding on the type semantics to obtain the semantic-based global features.

[0074] In this embodiment, the research is based on indoor three-dimensional scene point cloud data, supplemented by semantic and two-dimensional image data, which are processed and transformed to form different modal data, and these multimodal data are extracted by different methods at different stages to obtain key information about indoor scenes, aiming to make full use of the information of different modal data and provide an important data basis for subsequent indoor scene understanding. Figure 3 As shown in the figure, by extracting features from these multimodal data, key information about indoor scenes is obtained, and the information of different modal data is fully utilized to provide important data support for subsequent indoor scene understanding.

[0075] In this embodiment, semantic-based global features can be obtained by extracting semantic features. With the development of large language models, the application of semantic features has become more flexible. The study combines the semantic information of the data and obtains semantic-based global features through feature encoding as one of the optimization methods of the study. First, the original semantic information of the preprocessed point cloud data is obtained, and the original semantic information of the preprocessed point cloud data is semantically segmented and the corresponding sample labels are generated. The corresponding type semantics are extracted according to these sample labels, and the semantic features of these type semantics are encoded to obtain semantic-based global features.

[0076] Step S123, performing instance segmentation on the preprocessed point cloud data through the Transformer-based 3D semantic instance segmentation model, extracting a single instance, and obtaining the single instance global feature through instance feature encoding.

[0077] Step S124, extracting corresponding instance masks and corresponding semantic information according to the instance results obtained by segmentation using the Transformer-based 3D semantic instance segmentation model, and performing multi-view depth map projection according to the instance masks and the corresponding semantic information to obtain a multi-view depth map.

[0078] Step S125 , extracting multi-view features through a language-image comparison pre-training model, aggregating the multi-view features, and obtaining the multi-view fusion-based global features through fusion feature encoding.

[0079] In this embodiment, the global features of a single instance can be obtained by extracting features based on a single segmented instance. Based on the instance results of Mask3D segmentation, a single instance is extracted, and features are extracted from the instance through the Res16UNet34C network (a deep learning model that combines residual network and U-Net convolutional network structure) to obtain the global features of a single instance, providing data support for subsequent multimodal feature fusion.

[0080] In this embodiment, global features based on multi-view fusion can be obtained through feature extraction based on multi-view. CLIP is a powerful model that can understand image and text data at the same time. CLIP is used to extract multi-view 2D image features obtained by high-fidelity projection of instance results based on Mask3D segmentation, which can be used for text generation, semantic analysis, feature alignment fusion, etc., thereby achieving a better understanding of indoor scenes. According to the instance results obtained by segmentation based on the Transformer-based 3D semantic instance segmentation model, the corresponding instance Mask and the corresponding semantic information are extracted, and feature extraction and high-fidelity multi-view projection are performed on the instance query results to obtain two-dimensional images of several different perspectives and global features of a single instance. Multi-view feature extraction is performed through a language-image comparison pre-training model, and multi-view features are aggregated, and global features based on multi-view fusion are obtained through fusion feature encoding.

[0081] In this embodiment, the Res16UNet34C network, a feature backbone of the Transformer-based 3D semantic instance segmentation model, performs multi-resolution feature extraction on 3D point cloud data, and performs instance query analysis to obtain the corresponding instance Mask and corresponding semantic information; feature extraction and high-fidelity multi-view projection are performed on the instance query results to obtain several two-dimensional images of different perspectives and the global features of a single instance, and the CLIP network model is used to perform feature extraction on the multi-perspective depth image to obtain the global features of multi-perspective fusion.

[0082] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the method for indoor three-dimensional scene understanding by embedding multimodal features further includes the following steps:

[0083] Step S200: fusing the extracted multimodal features to obtain multimodal fusion features, and optimizing the multimodal feature embedded indoor three-dimensional scene understanding model through the multimodal fusion features.

[0084] In this embodiment, a comprehensive understanding of indoor three-dimensional scenes is achieved through multimodal feature fusion. This process includes text generation based on two-dimensional features and semantic information, and these features are mapped to the space described by natural language through a text generation model. Further, the text alignment generated by multimodal features ensures the consistency of different modal features, and the information of each modality is fully utilized through a fusion strategy. Finally, the model is iteratively trained again by using the fused and enhanced features, which improves the original model's ability to understand indoor three-dimensional scenes and achieves a comprehensive perception of scene structure, object position and semantic information. This process integrates multi-source data information and multimodal feature information, and improves the accuracy and robustness of scene understanding. In addition, a multimodal feature-embedded indoor three-dimensional scene understanding model that integrates Mask3D, CLIP and other models is designed and constructed, and the model is trained and optimized using comprehensive multimodal fusion features.

[0085] Specifically, in an implementation of this embodiment, step S200 includes the following steps:

[0086] Step S210, fusing the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature and the multi-view fusion-based global feature to obtain the multimodal fusion feature.

[0087] Step S220: retrain and adjust the parameters of the multimodal feature embedded indoor three-dimensional scene understanding model through the multimodal fusion features to optimize the model parameter configuration.

[0088] In this embodiment, optimization is performed at the feature level. The main optimization method is to integrate multimodal features based on point cloud and its variant data, including geometric feature information of point cloud, multi-view feature fusion information, multi-resolution feature information of 3D point cloud, two-dimensional image feature information and semantic information, so as to realize comprehensive indoor scene information collection; and based on the Transformer-based 3D semantic instance segmentation model, a feature extraction module of the CLIP network, semantic information optimization and feature fusion technologies are embedded, and multimodal feature extraction, multimodal feature fusion, model optimization based on fusion features, semantic feature extraction optimization and other methods are performed on two-dimensional images and point clouds, so as to realize semantic analysis and comprehensive understanding of indoor three-dimensional scenes.

[0089] Specifically, in an implementation of this embodiment, step S220 includes the following steps:

[0090] Step S221, performing text generation and text alignment operations on the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature to obtain aligned multimodal features.

[0091] Step S222, fusing the aligned multimodal features to obtain the multimodal fusion features.

[0092] In this embodiment, the extracted features can be converted into text descriptions. For example, for an object in an image, its shape, color, texture and other attributes can be described; for point cloud data, the object's position, size, direction and other information can be described. These text descriptions provide a basis for subsequent text processing and analysis. In order to enable features from different modalities to be associated and compared with each other, a text alignment operation is required to obtain aligned multimodal features, which provides a better basis for subsequent multimodal data processing and analysis.

[0093] In this embodiment, by fusing multimodal features such as 3D, 2D and text, the feature part of the Transformer-based 3D semantic instance segmentation model is optimized and enhanced, which can enhance its understanding of indoor 3D scenes, including more comprehensive perception of the scene structure, object location and semantic information. Through feature fusion, various information sources can be better integrated to improve the accuracy and robustness of scene understanding.

[0094] Specifically, in an implementation of this embodiment, step S221 includes the following steps:

[0095] Step S221a, through a text generation model, maps the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature and the multi-view fusion-based global feature to the space of natural language description to obtain a corresponding text description sequence.

[0096] Step S221b, performing a text alignment operation on the text description sequence, and analyzing and verifying the result of the text alignment to obtain the aligned multimodal features.

[0097] In this embodiment, the core purpose of text generation is to convert the above-extracted two-dimensional and three-dimensional features and semantic information into understandable natural language text, providing a data basis for subsequent research on text alignment and feature fusion. The main method is to map the two-dimensional and three-dimensional features to the space of natural language description through a text generation model, and then obtain a text description sequence consistent with the two-dimensional scene features. In order to fuse multi-resolution features based on 3D point clouds, multi-view 2D image features, text features, etc. to form a comprehensive multimodal feature representation, it is first necessary to align the text generated based on the features and align it with the real data set to verify its effectiveness. The process designs some fusion strategies such as weight distribution and normalization to ensure that features of different modalities are fully utilized.

[0098] In this embodiment, multimodal feature extraction and fusion training based on CLIP. By constructing the Realistic_Projection method (Realistic_Projection method is a technology for generating realistic projection), the instance results of Mask3D segmentation are projected with high fidelity, the point cloud data is converted into a multi-view depth image, and the multi-view depth map and two-dimensional scene map are extracted by CLIP, and the text is generated by combining semantic features and instance features, and text alignment is performed to achieve feature alignment fusion. Then, the fused and enhanced feature information is used for model training, which can better integrate various feature information and improve the accuracy and robustness of model scene understanding.

[0099] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the method for indoor three-dimensional scene understanding by embedding multimodal features further includes the following steps:

[0100] Step S300: extracting semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and outputting the optimized instance mask and the optimized semantic information.

[0101] In this embodiment, after the models are trained, the preprocessed point cloud data can be input into the optimized multimodal feature embedded indoor three-dimensional scene understanding model for feature extraction, and the semantic classification optimization results can be output.

[0102] Specifically, in an implementation of this embodiment, step S300 includes the following steps:

[0103] Step S310: extracting semantic features of the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model to obtain the optimized instance mask and the optimized semantic information corresponding to the preprocessed point cloud data.

[0104] Step S320: output the optimized instance mask and the optimized semantic information.

[0105] In this embodiment, semantic features are extracted from preprocessed point cloud data by using a multimodal feature embedded indoor 3D scene understanding model, and optimized semantic information and optimized mask information corresponding to the preprocessed point cloud data can be obtained, and these information are output as semantic classification optimization results. These results can be used to further analyze, understand and utilize the semantic information in the point cloud data, improve the accuracy and robustness of scene understanding, and provide support for intelligent perception and interaction of indoor scenes.

[0106] This embodiment achieves the following technical effects through the above technical solution:

[0107] This embodiment is based on the original point cloud data, and obtains 3D point cloud data that meets the requirements through preprocessing methods such as supervoxelization; then, based on the Transformer's 3D semantic instance segmentation model feature backbone Res16UNet34C network, multi-resolution feature extraction is performed on the 3D point cloud data, and instance query analysis is performed to obtain instance Mask and corresponding semantic information; feature extraction and high-fidelity multi-view projection are performed on the instance query results to obtain several two-dimensional images of different perspectives and global features of a single instance, and the multi-perspective depth image is specially extracted through the CLIP network model to obtain multi-perspective fused global features; combined with the extracted multimodal features and the real features of the data set, text generation and alignment are performed respectively with the aid of semantic information, and then the multimodal features are fused to achieve feature enhancement and make full use of data features of different modalities; finally, the fused multimodal features are used to optimize the indoor three-dimensional scene understanding model embedded with multimodal features of models such as Mask3D and CLIP, and a method for indoor three-dimensional scene understanding based on multimodal feature embedding is realized.

[0108] Exemplary Devices

[0109] Based on the above embodiments, the present invention also provides a terminal, comprising: a processor, a memory, an interface, a display screen and a communication module connected via a system bus; wherein the processor is used to provide computing and control capabilities; the memory includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0110] When the computer program is executed by the processor, it is used to implement the operation of a method for indoor three-dimensional scene understanding with multimodal feature embedding.

[0111] It can be understood by those skilled in the art that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] In one embodiment, a terminal is provided, comprising: a processor and a memory, wherein the memory stores a multimodal feature embedded indoor three-dimensional scene understanding program, and the multimodal feature embedded indoor three-dimensional scene understanding program is used to implement the operation of the multimodal feature embedded indoor three-dimensional scene understanding method as described above when executed by the processor.

[0113] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a multimodal feature embedded indoor three-dimensional scene understanding program, and when the multimodal feature embedded indoor three-dimensional scene understanding program is executed by the processor, it is used to implement the operation of the multimodal feature embedded indoor three-dimensional scene understanding method as described above.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory.

[0115] In summary, the present invention provides a method, terminal and storage medium for indoor three-dimensional scene understanding with multimodal feature embedding, the method comprising: acquiring a two-dimensional image and preprocessed point cloud data, and performing multimodal feature extraction on the two-dimensional image and the preprocessed point cloud data; fusing the extracted multimodal features to obtain multimodal fusion features, and optimizing a multimodal feature embedded indoor three-dimensional scene understanding model through the multimodal fusion features; extracting semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and outputting a semantic classification optimization result; the present invention proposes a new method for indoor three-dimensional scene understanding based on a Transformer-based 3D semantic instance segmentation model combined with multimodal features, aiming to improve the model's ability to understand indoor three-dimensional scenes, so that it can more accurately identify and classify each element in the scene, and enhance the accuracy and comprehensiveness of indoor three-dimensional scene understanding.

[0116] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for indoor three-dimensional scene understanding based on multimodal feature embedding, characterized in that: include: Acquire a two-dimensional image and preprocessed point cloud data, and perform multimodal feature extraction on the two-dimensional image and the preprocessed point cloud data; The extracted multimodal features are fused to obtain multimodal fusion features, and the multimodal feature embedded indoor three-dimensional scene understanding model is optimized by the multimodal fusion features; Extracting semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and outputting optimized instance masks and optimized semantic information; The acquiring of the two-dimensional image and the pre-processed point cloud data, and performing multimodal feature extraction on the two-dimensional image and the pre-processed point cloud data, includes: The two-dimensional image is acquired, a mask corresponding to a single target is obtained through target recognition and extraction, and visual feature extraction is performed through a language-image comparison pre-training model to obtain a two-dimensional single target global feature; Acquire the preprocessed point cloud data, perform multimodal feature extraction on the preprocessed point cloud data, and perform instance segmentation through a 3D semantic instance segmentation model based on deep learning to obtain semantic global features, single instance global features, and multi-view fusion global features; The multimodal feature extraction is performed on the preprocessed point cloud data, and instance segmentation is performed through a 3D semantic instance segmentation model based on deep learning to obtain semantic global features, single instance global features, and multi-view fusion global features, including: Performing semantic segmentation on the preprocessed point cloud data and generating corresponding sample labels, extracting corresponding type semantics according to the sample labels, and performing semantic feature encoding on the type semantics to obtain the semantic-based global features; Performing instance segmentation on the preprocessed point cloud data by using the deep learning-based 3D semantic instance segmentation model, extracting a single instance, and obtaining the single instance global feature by encoding the instance feature; According to the instance result obtained by segmentation of the 3D semantic instance segmentation model based on deep learning, a corresponding instance mask and corresponding semantic information are extracted, and a multi-view depth map is projected according to the instance mask and the corresponding semantic information to obtain a multi-view depth map; Extract multi-view features through a language-image comparison pre-training model, aggregate the multi-view features, and obtain the multi-view fusion-based global features through fusion feature encoding; The extracted multimodal features are fused to obtain multimodal fusion features, and the multimodal feature embedded indoor three-dimensional scene understanding model is optimized by the multimodal fusion features, including: The two-dimensional single target global feature, the semantic-based global feature, the single instance global feature and the multi-view fusion-based global feature are fused to obtain the multimodal fusion feature; Retraining and adjusting the parameters of the multimodal feature embedded indoor three-dimensional scene understanding model through the multimodal fusion features to optimize the model parameter configuration; The fusing the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature and the multi-view fusion-based global feature to obtain the multimodal fusion feature includes: Performing text generation and text alignment operations on the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature to obtain aligned multimodal features; The aligned multimodal features are fused to obtain the multimodal fusion features.

2. The method for indoor three-dimensional scene understanding based on multimodal feature embedding according to claim 1, characterized in that: The obtaining of the preprocessed point cloud data comprises: Acquire point cloud data, perform quantization, densification, smoothing, compression and supervoxelization operations on the point cloud data, and obtain the preprocessed point cloud data.

3. The method for indoor three-dimensional scene understanding based on multimodal feature embedding according to claim 1, characterized in that: The operations of performing text generation and text alignment on the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature to obtain aligned multimodal features include: Through a text generation model, the two-dimensional single target global feature, the semantic-based global feature, the single instance global feature, and the multi-view fusion-based global feature are mapped to a natural language description space to obtain a corresponding text description sequence; A text alignment operation is performed on the text description sequence, and the result of the text alignment is analyzed and verified to obtain the aligned multimodal features.

4. The method for indoor three-dimensional scene understanding based on multimodal feature embedding according to claim 1, characterized in that: The extracting of semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model, and outputting the optimized instance mask and the optimized semantic information, comprises: Extracting semantic features from the preprocessed point cloud data according to the multimodal feature embedded indoor three-dimensional scene understanding model to obtain the optimized instance mask and the optimized semantic information corresponding to the preprocessed point cloud data; The optimized instance mask and the optimized semantic information are output.

5. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a multimodal feature embedded indoor three-dimensional scene understanding program, and when the multimodal feature embedded indoor three-dimensional scene understanding program is executed by the processor, it is used to implement the operation of the multimodal feature embedded indoor three-dimensional scene understanding method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal feature embedded indoor three-dimensional scene understanding program, which, when executed by a processor, is used to implement the operation of the multimodal feature embedded indoor three-dimensional scene understanding method as described in any one of claims 1 to 4.

Images