Three-dimensional scene atlas room classification method and system based on multi-modal feature tensor
By using multimodal feature tensor decomposition and the PARAFAC2 model, the number of room categories is automatically determined and semantic labels are assigned, which solves the problem of multimodal fusion and automatic discovery in room classification in existing technologies, and improves classification accuracy and robot navigation efficiency.
Patent Information
- Application Number
- CN202511585646.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-02-27
AI Technical Summary
Existing technologies lack solutions for room classification that can systematically integrate multimodal heterogeneous features and automatically discover the number of real room categories in the environment, resulting in limited classification capabilities, poor robustness, and reliance on manual prior knowledge or large amounts of labeled data.
By using multimodal feature tensor decomposition, feature matrices are extracted using different pre-trained models. The PARAFAC2 decomposition model is then used to decompose the feature matrices into observation-class correlation matrices and feature-class factor matrices, automatically determining the number of room categories. Semantic category information is then assigned to the room layer nodes through semantic similarity matching.
It achieves automated and interpretable room classification, improves the accuracy and robustness of classification, reduces the dependence on expert knowledge and labeled data, enhances the semantic richness and machine readability of 3D scene maps, and supports high-level semantic navigation and task planning for robots.
Smart Images
Figure CN121582631A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically to a method and system for classifying rooms in a 3D scene map based on multimodal feature tensors. Background Technology
[0002] With the rapid development of service robots, augmented reality, and other fields, the perception needs of intelligent agents for indoor environments have shifted from traditional geometric maps to semantic environment models capable of supporting high-level semantic reasoning. 3D scene graphs, as a core representation technology, organically integrate geometric and semantic information through a graph structure of nodes and edges. The room layer, connecting the object layer and the building layer, is crucial for bridging the gap between object perception and scene understanding. However, achieving automated and high-precision room classification remains a challenging problem in the field of environmental perception.
[0003] Existing technical solutions have obvious limitations: methods based on geometry and topology completely ignore the semantic content inside the room and have limited classification capabilities; methods based on object co-occurrence rules and knowledge graphs rely on manual prior rules, have poor robustness, and are difficult to cover complex scenarios; semantic reasoning methods based on large language models get rid of explicit rules, but suffer from problems such as illusion phenomena, discarding visual features, and black boxes in the decision-making process; and methods based on traditional machine learning and deep learning heavily rely on large amounts of labeled data and have insufficient generalization capabilities.
[0004] The core bottleneck of current technology lies in the lack of a solution capable of systematically integrating multimodal heterogeneous features and automatically discovering the actual number of room categories in the environment. Existing methods either rely on fragile human priors, are limited by a single source of information, or have uninterpretable decision-making processes that depend on large amounts of labeled data. Therefore, there is an urgent need in this field for a new technical solution that can fully utilize the complementary information provided by multiple models to solve the room classification problem transparently and robustly in a data-driven manner. Summary of the Invention
[0005] To address the aforementioned issues, this invention provides a method and system for classifying rooms in a three-dimensional scene atlas based on multimodal feature tensors.
[0006] In a first aspect, the technical solution of the present invention provides a method for room classification of three-dimensional scene maps based on multimodal feature tensor decomposition, comprising the following steps: S1, Through Several different pre-trained models were used to collect data from the environment. Feature extraction is performed on scene observation data, and the extracted feature vectors are organized according to the model. The set of characteristic matrices is the set of characteristic matrices. S2. Perform tensor decomposition on the feature matrix set to simultaneously obtain: a common observation-class correlation matrix, which represents the weight of each observation belonging to each potential room class; and a feature-class factor matrix corresponding to each model, where the column vectors of the matrix represent the prototype features of the potential room class in the model feature space; wherein, the rank of the tensor decomposition is determined as the number of potential room classes in the environment. S3. For each potential room category, the prototype features in each feature-category factor matrix are fused to form a fused category prototype representation; by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label, the most matching room type label is assigned to each potential room category. S4. Determine the room category to which each scene observation belongs based on the observation-category correlation matrix, and assign the corresponding room type label to the room layer node in the 3D scene atlas, thereby adding semantic category information to the room layer.
[0007] The observation-category correlation matrix and feature-category factor matrix obtained by decomposition have clear physical meanings, clearly revealing the category assignment probability of each observation and the prototype features of each category, making the classification decision process no longer a black box, and facilitating system debugging and result verification.
[0008] By integrating automatically discovered and accurately labeled room category information into the 3D scene atlas, the semantic richness and machine readability of the atlas are greatly enhanced, providing a reliable environmental cognition foundation for robot high-level semantic navigation and task planning.
[0009] Feature extraction is performed on scene observation data using multiple pre-trained models, and the extracted features are organized into a feature matrix set. Next, tensor decomposition is performed on this set, simultaneously obtaining a shared observation-class correlation matrix and a feature-class factor matrix for each model. The rank of the decomposition is automatically determined to represent the number of room categories. Then, prototype features from each factor matrix are fused to form a category prototype representation. By calculating its semantic similarity to predefined room type labels, the most matching semantic label is assigned to each potential category. Finally, the room category to which each observation belongs is determined based on the observation-class correlation matrix, and the corresponding type label is assigned to the room layer nodes in the 3D scene atlas, thus completing the automated and interpretable construction of room layer semantic information.
[0010] As a further limitation of the technical solution of the present invention, step S1 includes: S11. Control the intelligent agent equipped with sensors to move in the target environment, and collect data through the sensors. Observation data for each scene; S12. Input the observation data for each scene into... In each of the different pre-trained models, the input observation data is processed independently and a fixed-dimensional feature vector is output; the dimensions of the feature vectors output by different pre-trained models are allowed to be different. S13, the first Each pre-trained model for all The feature vectors extracted from the observed data are stacked as row vectors in transpose form to form the feature matrix corresponding to the pre-trained model. ; S14, All Feature matrices of pre-trained models This constitutes a set of feature matrices.
[0011] The complete technology chain from data acquisition and feature extraction to feature matrix organization was clearly defined, ensuring that multimodal features could be prepared in a standardized and efficient manner, providing high-quality and structured input for subsequent tensor decomposition.
[0012] As a further limitation of the technical solution of the present invention, in S2, the step of tensor decomposition of the feature matrix set includes: The set of feature matrices is modeled as an irregular three-dimensional tensor and input into the PARAFAC2 decomposition model for fitting. Each feature matrix is decomposed using PARAFAC2. It is decomposed into the product of three factor matrices, where the resulting factor matrices include: An observation-class correlation matrix shared across all models is used to represent the membership relationships between different scene observations and potential room classes; A set of model-related feature-category factor matrices, each matrix representing the prototype features of potential room categories in the model feature space; A factorization matrix.
[0013] By employing the PARAFAC2 decomposition model, this method can directly process irregular tensors composed of feature matrices of different dimensions. This characteristic allows the invention to flexibly integrate various heterogeneous feature extraction models without the need for cumbersome feature dimension alignment preprocessing. This maintains the integrity of the original feature information and enhances the versatility and flexibility of the method.
[0014] As a further limitation of the technical solution of the present invention, in S2, the step of determining the rank of the tensor decomposition as the number of potential room categories in the environment includes: Calculate at different ranks Model reconstruction residuals under PARAFAC2 decomposition; Plotting the reconstructed residuals with rank The curve of value change; Select the rank corresponding to the inflection point on the curve where the decreasing trend of the reconstructed residual changes. The value represents the number of potential room categories in the environment.
[0015] By analyzing the inflection point of the reconstructed residual curve, this method avoids the subjectivity and arbitrariness of manually selecting the number of categories, making the category discovery process completely data-driven, and the results more reliable and accurate.
[0016] As a further limitation of the technical solution of the present invention, the reconstruction residual is calculated by the following formula:
[0017] in, Describe the Frobenius norm. For a shared observation-category correlation matrix, For the first The feature-category factor matrix corresponding to each model For the first The decomposition factor matrix corresponding to each model.
[0018] As a further limitation of the technical solution of the present invention, in S3, the step of fusing the prototype features in each feature-category factor matrix to form a fused category prototype representation for each potential room category includes: For the For each potential room category, extract the corresponding _th ... Column vector; By using vector concatenation operations, the vectors from... The column vectors of each model are combined into a fused class prototype representation vector.
[0019] By using the simple yet effective operation of vector concatenation, prototype features from different models, reflecting different aspects of the same room category, are integrated into a unified and more informative representation. This fusion method preserves the feature information of each model to the greatest extent, laying the foundation for accurate semantic label mapping in the future.
[0020] As a further limitation of the technical solution of the present invention, in S3, the step of assigning the most matching room type label to each potential room category by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label includes: Predefine a set of room type labels; use Each pre-trained model has a corresponding text encoder that encodes each room type label to obtain a text feature vector for each model. For each room type label, all text feature vectors are fused to form a fused label semantic representation vector; Calculate the cosine similarity between the fused prototype representation vector of each potential room category and the semantic representation vectors of all fused labels; Assign the room type label with the highest cosine similarity to the potential room category.
[0021] By utilizing a text encoder derived from visual features and performing similarity calculations within the same multimodal semantic space, effective and accurate matching between data-driven category prototypes and knowledge-driven text labels is ensured. Furthermore, by fusing text features from multiple models to generate semantic representations of labels, the semantic understanding of the same label from different models is integrated, resulting in more robust and accurate final label assignment.
[0022] As a further limitation of the technical solution of the present invention, step S4 includes: S41. For the observation-category correlation matrix, determine the first... The column index of the largest numerical element in the row vector; S42. Determine the potential room category corresponding to the column index as the first... The final room category to which each scene observation belongs; S43, the first The room type label of the final room category to which each scene observation belongs is assigned to the room layer node in the 3D scene atlas associated with that observation; In the hierarchical structure of the 3D scene graph, the room layer node is located above the object layer node and is used to organize multiple object layer nodes within the spatial coverage area.
[0023] By assigning classification labels to room layer nodes located above the object layer, this method successfully elevates low-level perceptual information to high-level semantic concepts, constructing a machine-understandable and usable structured environment model that directly supports semantic-based robot task execution.
[0024] As a further limitation of the technical solution of the present invention, the three-dimensional scene map includes room layer nodes with room type labels, and the three-dimensional scene map is used for high-level semantic navigation and task planning of intelligent robots.
[0025] Atlases with accurate room category labels enable robots to truly understand the spatial semantics in high-level instructions, improving the success rate and efficiency of task planning.
[0026] Secondly, the technical solution of the present invention provides a three-dimensional scene atlas room classification system based on multimodal feature tensor decomposition, comprising: The data acquisition module is configured to control the movement of a sensor-equipped intelligent agent in a target environment, and to collect data through the sensors. Observation data for each scene; Feature extraction module, configured to be used to extract features A different pre-trained model for the Feature extraction is performed on scene observation data, and the extracted feature vectors are organized according to the model. The set of characteristic matrices is the set of characteristic matrices. The tensor decomposition module is configured to perform tensor decomposition on the feature matrix set to simultaneously obtain: a common observation-class correlation matrix, which represents the weight of each observation belonging to each potential room class; and a feature-class factor matrix corresponding to each model, wherein the column vectors of the matrix represent the prototype features of the potential room class in the model feature space; wherein the rank of the tensor decomposition is determined as the number of potential room classes in the environment. The semantic mapping module is configured to fuse prototype features in each feature-category factor matrix for each potential room category to form a fused category prototype representation; by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label, the most matching room type label is assigned to each potential room category. The map update module is configured to determine the room category to which each scene observation belongs based on the observation-category association matrix, and assign the corresponding room type label to the room layer node in the 3D scene map, thereby adding semantic category information to the room layer.
[0027] As a further limitation of the technical solution of the present invention, the feature extraction module is specifically configured to: Input the observation data for each scene separately to In each of the different pre-trained models, the input observation data is processed independently and a fixed-dimensional feature vector is output; the dimensions of the feature vectors output by different pre-trained models are allowed to be different. The first Each pre-trained model for all The feature vectors extracted from the observation data are stacked as row vectors in transpose form to form the feature matrix corresponding to the pre-trained model. All The feature matrices of each pre-trained model constitute a feature matrix set.
[0028] As a further limitation of the technical solution of the present invention, the tensor decomposition module is specifically configured for: The set of feature matrices is modeled as an irregular three-dimensional tensor and input into the PARAFAC2 decomposition model for fitting. Each feature matrix is decomposed into the product of three factor matrices using PARAFAC2 decomposition. The resulting factor matrices include: An observation-class correlation matrix shared across all models; A set of feature-category factor matrices related to the model; A factorization matrix.
[0029] As a further limitation of the technical solution of the present invention, the tensor decomposition module is also configured to: Calculate at different ranks Model reconstruction residuals under PARAFAC2 decomposition; Plotting the reconstructed residuals with rank The curve of value change; Select the rank corresponding to the inflection point on the curve where the decreasing trend of the reconstructed residual changes. The value represents the number of potential room categories in the environment.
[0030] As a further limitation of the technical solution of the present invention, the semantic mapping module is specifically configured to: For the For each potential room category, extract the corresponding _th ... Column vector; By using vector concatenation operations, the vectors from... The column vectors of each model are combined into a fused class prototype representation vector; use Each pre-trained model has a corresponding text encoder that encodes each room type label to obtain a text feature vector for each model. For each room type label, all text feature vectors are fused to form a fused label semantic representation vector; Calculate the cosine similarity between the fused prototype representation vector of each potential room category and the semantic representation vectors of all fused labels; Assign the room type label with the highest cosine similarity to the potential room category.
[0031] As a further limitation of the technical solution of the present invention, the map update module is specifically configured to: For the observation-category correlation matrix, determine the first... The column index of the largest numerical element in the row vector; The potential room category corresponding to the column index is determined as the first... The final room category to which each scene observation belongs; The first The room type label of the final room category to which each scene observation belongs is assigned to the room layer node in the 3D scene atlas associated with that observation.
[0032] As can be seen from the above technical solutions, this application has the following advantages: By constructing a feature matrix set and performing joint tensor decomposition, it can systematically integrate heterogeneous features from different pre-trained models, overcoming the limitations of existing methods that only utilize single-modal information, making the classification criteria more comprehensive and improving the classification accuracy and robustness in complex scenarios. By interpreting the rank of tensor decomposition as the number of room categories, it can automatically infer the potential room categories existing in the environment from the data without relying on manually predefined rules or a large amount of labeled data, reducing the dependence on expert knowledge and making the system highly adaptive. Attached Figure Description
[0033] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.
[0035] Figure 2 This is a schematic diagram of the decomposition of PARAFAC2.
[0036] Figure 3 A block diagram of a system provided in an embodiment of the present invention. Detailed Implementation
[0037] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0038] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0039] The implementation of this invention requires an intelligent agent device with autonomous mobility. The intelligent agent device is equipped with a computing unit and sensors such as a visual sensor, a lidar sensor, and an IMU. The computing unit has a basic operating system and is pre-configured with an environment for arbitrary independent lidar mapping and localization or multi-sensor fusion mapping and localization technology, an object and text recognition technology environment, and usable large language models and visual language model interfaces, thereby possessing the ability to construct 3D scene maps.
[0040] For unknown environments, a multi-layered global 3D scene atlas can be constructed, capable of modeling complex scenes from different abstract levels, from individual objects to entire buildings. This hierarchical structure not only provides rich semantic information but also allows for flexible adjustment of the representation's volume and detail according to different task requirements, thereby achieving comprehensive perception of indoor scenes. The 3D scene atlas includes a base layer, object layer, room layer, floor layer, and building layer. The base layer exists in the form of a 3D mesh, containing key metric-semantic information, providing basic support for the construction and navigation tasks of the upper layers. The object layer generates accurate geometric and semantic representations for each object instance through semantic segmentation and point cloud extraction. The room layer, floor layer, and building layer further organize the lower-level nodes to form a larger-scale spatial structure, supporting navigation and task planning for multi-region, multi-floor, or large buildings. Current 3D scene atlases already have layers below the room region layer (i.e., the first mesh layer and the second object layer). The room region layer is constructed using the technology proposed in this invention. Specifically, as shown... Figure 1 As shown, this embodiment of the invention provides a method for room classification of 3D scene maps based on multimodal feature tensor decomposition, including the following steps: S1, Through Several different pre-trained models were used to collect data from the environment. Feature extraction is performed on scene observation data, and the extracted feature vectors are organized according to the model. The set of characteristic matrices is obtained by this step; specifically, this step includes: Select Several different pre-trained feature extraction models were used (e.g., CLIP (Constastive Language-Image Pretraining) and DINOS (DETR with Improved deNoising anchOr boxes) with different parameter settings, such as ViT (Vision Transformer)-B / 14, ViT-B / 16, ViT-B / 32, etc.). The visual language pre-training model was the CLIP model, and the self-supervised visual feature extraction model was the DINO model.
[0041] The intelligent agent device traverses the entire scene and collects data. The first observation. For the first... Observation ( ), using the first One model ( ),calculate 3D feature vector .
[0042] All The first observation at the The feature vectors of each model are in the form of row vectors (i.e.) Stacked together, forming the first Room feature matrix under each model ;
[0043] Due to the different feature dimensions of the models It might be different; here we don't construct a regular three-dimensional tensor (e.g.) (a three-dimensional tensor), but rather... Room feature matrix of each model It can be viewed as an irregular three-dimensional tensor or tensor sequence. This serves as the input for the subsequent PARAFAC2 (parallel factor analysis 2) decomposition. This three-dimensional tensor... have The slice, as the third dimension. The dimensions of the film are (like Figure 2 (As an example).
[0044] This approach achieves true deep fusion and collaborative utilization of multimodal information, overcoming the limitations of existing methods that rely solely on single-modal information. It jointly analyzes and optimizes features from multiple models (such as visual encoders and semantic models) across different architectures and modalities within a unified mathematical framework (tensor). By fully leveraging the complementarity of visual and semantic features, it provides more comprehensive and richer classification criteria, improving classification accuracy and robustness, especially in complex scenarios with blurred object appearances, poor lighting, or ambiguous text descriptions.
[0045] S2. Perform tensor decomposition on the feature matrix set to simultaneously obtain: a common observation-class correlation matrix, which represents the weight of each observation belonging to each potential room class; and a feature-class factor matrix corresponding to each model, where the column vectors of the matrix represent the prototype features of the potential room class in the model feature space; wherein, the rank of the tensor decomposition is determined as the number of potential room classes in the environment. The steps for tensor decomposition of the feature matrix set in S2 include: The set of feature matrices is modeled as an irregular three-dimensional tensor and input into the PARAFAC2 decomposition model for fitting. Each feature matrix is decomposed using PARAFAC2. It is decomposed into the product of three factor matrices, where the resulting factor matrices include: An observation-class correlation matrix shared across all models is used to represent the membership relationships between different scene observations and potential room classes; A set of model-related feature-category factor matrices, each matrix representing the prototype features of a potential room category in a specific model feature space; A factorization matrix; Specifically, the feature matrix set is input into the PARAFAC2 decomposition model; through the PARAFAC2 decomposition model, each feature matrix is decomposed... Decomposed into three factor matrices, such that , , A shared observation-category correlation matrix; The Middle Line number Column elements Indicates the first The scene observation belongs to the first The probability of each potential room category For the first Each model corresponds to a feature-category factor matrix; each column of which... Representing the The room area in the The prototype features in the feature space of the model, i.e. the first... The visual characteristics of each room area in the first A highly condensed model. For the first The factorization matrix corresponding to each model is a a diagonal matrix; To decompose the rank, reveal the number of scene regions corresponding to all observations; The PARAFAC2 decomposition is solved using algorithms such as alternating least squares (ALS).
[0046] The steps for determining the rank of the tensor decomposition in S2 as the number of potential room categories in the environment include: Calculate at different ranks Model reconstruction residuals under PARAFAC2 decomposition; Plotting the reconstructed residuals with rank The curve of value change; Select the rank corresponding to the inflection point on the curve where the decreasing trend of the reconstructed residual changes. The value represents the number of potential room categories in the environment.
[0047] The reconstruction residuals from the PARAFAC2 decomposition are calculated as follows:
[0048] in, Describe the Frobenius norm. For a shared observation-category correlation matrix, For the first The feature-category factor matrix corresponding to each model For the first The decomposition factor matrix corresponding to each model.
[0049] By traversing different Analyze the values and observe the decreasing curve of the model residuals, and select an optimal one. Value. Residual follows The inflection point of increase followed by decrease corresponds to the number of room categories closest to reality.
[0050] It provides data-driven and automated room category discovery capabilities, without relying on manually predefined rules or large amounts of labeled data. Through tensor decomposition, it can automatically infer the number of potential room categories in a scene (i.e., the rank of the decomposition) directly from the input multi-model feature data. It reduces reliance on expert knowledge and labeled data, and the system is highly adaptable, able to automatically adapt to new and unseen environmental layouts and room function combinations, achieving a higher degree of automation.
[0051] S3. For each potential room category, the prototype features in each feature-category factor matrix are fused to form a fused category prototype representation; by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label, the most matching room type label is assigned to each potential room category. In this step, for each potential room category, the steps of fusing the prototype features in each feature-category factor matrix to form the fused category prototype representation include: For the 1 potential room category, from each feature-category factor matrix Extract the corresponding first column vector ; By using vector concatenation operations, the vectors from... Column vectors of each model Combined into a fused category prototype representation vector ; For each potential room area category ,Depend on The fusion feature-class vector, jointly defined by all models, is calculated as follows:
[0052] in, It is a fusion function that concatenates input vectors.
[0053] In this step, the process of assigning the most matching room type label to each potential room category by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label includes: Predefined set of room type labels (e.g., {kitchen, bedroom, living room, bathroom, office, ...}); use Each pre-trained model corresponds to a text encoder, for each room type label. Encode the text to obtain text feature vectors for each model. ; For each room type label All text feature vectors are merged to form a fused label semantic representation vector. ;Depend on The fusion room category label text feature vector, jointly defined by the models, is calculated as follows:
[0054] Calculate the fusion prototype representation vector for each potential room category With all fused label semantic representation vectors Cosine similarity between ;
[0055] Assign the room type label with the highest cosine similarity to the potential room category. ,Right now .
[0056] S4. Determine the room category to which each scene observation belongs based on the observation-category correlation matrix, and assign the corresponding room type label to the room layer node in the 3D scene atlas, thereby adding semantic category information to the room layer.
[0057] Finally, for each observation (the first one) (one observation), based on the PARAFAC2 factor matrix The Middle The maximum index of the row is used to determine the category to which the observation belongs, and the corresponding room area category label is assigned. Correspondingly, with the first The second-layer object node corresponding to each observation belongs to the room area. Therefore, the aforementioned room area classification module can be integrated into the 3D scene atlas construction process to generate room layer nodes. Specifically, this step includes: S41. For the observation-category correlation matrix, determine the first... The column index of the largest numerical element in the row vector ; S42, Index the column The corresponding potential room category was determined to be No. The final room category to which each scene observation belongs; S43, the first The room type label of the final room category to which each scene observation belongs is assigned to the room layer node in the 3D scene atlas associated with that observation; In the hierarchical structure of the 3D scene graph, the room layer node is located above the object layer node and is used to organize multiple object layer nodes within the spatial coverage area.
[0058] It should be noted that the three-dimensional scene map includes room layer nodes with room type labels, and the three-dimensional scene map is used for high-level semantic navigation and task planning of intelligent robots.
[0059] The factor matrices generated by tensor decomposition have clear physical meanings; the room-category association matrix reveals the probability of each room belonging to each category; and the feature-category factor matrix provides the prototype features of each category. The classification decision process is no longer a black box. It can clearly explain why a room is assigned to a certain category (based on its similarity to the prototype features) and what the core features of each category are. This greatly facilitates system debugging, result verification, and the establishment of user trust.
[0060] When applied, this technology can strongly complement and enhance existing mainstream technologies (LLM). The output of this application (such as the determined number of categories and category prototype features) can serve as accurate prior knowledge to construct more reliable LLM prompts and guide its reasoning. A hybrid system can be constructed, utilizing this application for stable and interpretable coarse classification or category number estimation, and then using LLM for fine-grained description generation or complex scene reasoning, thereby combining the advantages of both to build a more powerful and robust room understanding system.
[0061] This application improves the semantic quality of 3D scene maps and the performance of robot tasks by providing accurate and structured room category information, thereby enhancing the semantic richness and machine readability of the 3D scene maps. Based on accurate room labels, robots can quickly and reliably perform tasks requiring high-level spatial semantic understanding (such as "go to the bedroom to get your glasses"), improving the success rate and efficiency of task planning and promoting the development of truly intelligent robot navigation and interaction.
[0062] like Figure 3 As shown, this embodiment of the invention provides a 3D scene atlas room classification system based on multimodal feature tensor decomposition, including: The data acquisition module is configured to control the movement of a sensor-equipped intelligent agent in a target environment, and to collect data through the sensors. Observation data for each scene; Feature extraction module, configured to be used to extract features A different pre-trained model for the Feature extraction is performed on scene observation data, and the extracted feature vectors are organized according to the model. The set of characteristic matrices is the set of characteristic matrices. The tensor decomposition module is configured to perform tensor decomposition on the feature matrix set to simultaneously obtain: a common observation-class correlation matrix, which represents the weight of each observation belonging to each potential room class; and a feature-class factor matrix corresponding to each model, wherein the column vectors of the matrix represent the prototype features of the potential room class in the model feature space; wherein the rank of the tensor decomposition is determined as the number of potential room classes in the environment. The semantic mapping module is configured to fuse prototype features in each feature-category factor matrix for each potential room category to form a fused category prototype representation; by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label, the most matching room type label is assigned to each potential room category. The map update module is configured to determine the room category to which each scene observation belongs based on the observation-category association matrix, and assign the corresponding room type label to the room layer node in the 3D scene map, thereby adding semantic category information to the room layer.
[0063] In some embodiments, the feature extraction module is specifically configured to: Input the observation data for each scene separately to In each of the different pre-trained models, the input observation data is processed independently and a fixed-dimensional feature vector is output; the dimensions of the feature vectors output by different pre-trained models are allowed to be different. The first Each pre-trained model for all The feature vectors extracted from the observation data are stacked as row vectors in transpose form to form the feature matrix corresponding to the pre-trained model. All The feature matrices of each pre-trained model constitute a feature matrix set.
[0064] In some embodiments, the tensor decomposition module is specifically configured to: The set of feature matrices is modeled as an irregular three-dimensional tensor and input into the PARAFAC2 decomposition model for fitting. Each feature matrix is decomposed into the product of three factor matrices using PARAFAC2 decomposition. The resulting factor matrices include: An observation-class correlation matrix shared across all models; A set of feature-category factor matrices related to the model; A factorization matrix.
[0065] In some embodiments, the tensor decomposition module is further configured to: Calculate at different ranks Model reconstruction residuals under PARAFAC2 decomposition; Plotting the reconstructed residuals with rank The curve of value change; Select the rank corresponding to the inflection point on the curve where the decreasing trend of the reconstructed residual changes. The value represents the number of potential room categories in the environment.
[0066] In some embodiments, the semantic mapping module is specifically configured to: For the For each potential room category, extract the corresponding _th ... Column vector; By using vector concatenation operations, the vectors from... The column vectors of each model are combined into a fused class prototype representation vector; use Each pre-trained model has a corresponding text encoder that encodes each room type label to obtain a text feature vector for each model. For each room type label, all text feature vectors are fused to form a fused label semantic representation vector; Calculate the cosine similarity between the fused prototype representation vector of each potential room category and the semantic representation vectors of all fused labels; Assign the room type label with the highest cosine similarity to the potential room category.
[0067] In some embodiments, the map update module is specifically configured to: For the observation-category correlation matrix, determine the first... The column index of the largest numerical element in the row vector; The potential room category corresponding to the column index is determined as the first... The final room category to which each scene observation belongs; The first The room type label of the final room category to which each scene observation belongs is assigned to the room layer node in the 3D scene atlas associated with that observation.
[0068] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for room classification of 3D scene maps based on multimodal feature tensor decomposition, characterized in that, Includes the following steps: S1, Through Several different pre-trained models were used to collect data from the environment. Feature extraction is performed on scene observation data, and the extracted feature vectors are organized according to the model. The set of characteristic matrices is the set of characteristic matrices. S2. Perform tensor decomposition on the feature matrix set to simultaneously obtain: a common observation-class correlation matrix, which represents the weight of each observation belonging to each potential room class; and a feature-class factor matrix corresponding to each model, where the column vectors of the matrix represent the prototype features of the potential room class in the model feature space; wherein, the rank of the tensor decomposition is determined as the number of potential room classes in the environment. S3. For each potential room category, the prototype features in each feature-category factor matrix are fused to form a fused category prototype representation; by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label, the most matching room type label is assigned to each potential room category. S4. Determine the room category to which each scene observation belongs based on the observation-category correlation matrix, and assign the corresponding room type label to the room layer node in the 3D scene atlas, thereby adding semantic category information to the room layer.
2. The method for room classification of 3D scene atlas based on multimodal feature tensor decomposition according to claim 1, characterized in that, The steps in S1 include: S11. Control the intelligent agent equipped with sensors to move in the target environment, and collect data through the sensors. Observation data for each scene; S12. Input the observation data for each scene separately to... In each of the different pre-trained models, the input observation data is processed independently and a fixed-dimensional feature vector is output; the dimensions of the feature vectors output by different pre-trained models are allowed to be different. S13, the first Each pre-trained model for all The feature vectors extracted from the observed data are stacked as row vectors in transpose form to form the feature matrix corresponding to the pre-trained model. ; S14, All Feature matrices of pre-trained models This constitutes a set of feature matrices.
3. The method for room classification of 3D scene atlas based on multimodal feature tensor decomposition according to claim 2, characterized in that, In S2, the steps for tensor decomposition of the feature matrix set include: The set of feature matrices is modeled as an irregular three-dimensional tensor and input into the PARAFAC2 decomposition model for fitting. Each feature matrix is decomposed using PARAFAC2. It is decomposed into the product of three factor matrices, where the resulting factor matrices include: An observation-class correlation matrix shared across all models is used to represent the membership relationships between different scene observations and potential room classes; A set of model-related feature-category factor matrices, each matrix representing the prototype features of potential room categories in the model feature space; A factorization matrix.
4. The method for room classification of 3D scene maps based on multimodal feature tensor decomposition according to claim 3, characterized in that, In S2, the steps for determining the rank of the tensor decomposition as the number of potential room categories in the environment include: Calculate at different ranks Model reconstruction residuals from PARAFAC2 decomposition under the given value; Plotting the reconstructed residuals with rank A curve showing the change in value; Select the rank corresponding to the inflection point on the curve where the decreasing trend of the reconstructed residual changes. The value represents the number of potential room categories in the environment.
5. The method for room classification of 3D scene atlas based on multimodal feature tensor decomposition according to claim 4, characterized in that, The reconstruction residuals are calculated using the following formula: in, Describe the Frobenius norm. For a shared observation-category correlation matrix, For the first The feature-category factor matrix corresponding to each model For the first The decomposition factor matrix corresponding to each model.
6. The method for room classification of three-dimensional scene maps based on multimodal feature tensor decomposition according to claim 5, characterized in that, In S3, for each potential room category, the steps to fuse the prototype features in each feature-category factor matrix to form a fused category prototype representation include: For the For each potential room category, extract the corresponding _th ... Column vector; By using vector concatenation operations, the vectors from... The column vectors of each model are combined into a fused class prototype representation vector.
7. The method for room classification of 3D scene atlas based on multimodal feature tensor decomposition according to claim 6, characterized in that, In S3, the step of assigning the most matching room type label to each potential room category by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label includes: Predefine a set of room type labels; use Each pre-trained model has a corresponding text encoder that encodes each room type label to obtain a text feature vector for each model. For each room type label, all text feature vectors are fused to form a fused label semantic representation vector; Calculate the cosine similarity between the fused prototype representation vector of each potential room category and the semantic representation vectors of all fused labels; Assign the room type label with the highest cosine similarity to the potential room category.
8. The method for room classification of 3D scene atlas based on multimodal feature tensor decomposition according to claim 7, characterized in that, The steps in S4 include: S41. For the observation-category correlation matrix, determine the first... The column index of the largest numerical element in the row vector; S42. Determine the potential room category corresponding to the column index as the first... The final room category to which each scene observation belongs; S43, the first The room type label of the final room category to which each scene observation belongs is assigned to the room layer node in the 3D scene atlas associated with that observation; In the hierarchical structure of the 3D scene graph, the room layer node is located above the object layer node and is used to organize multiple object layer nodes within the spatial coverage area.
9. The method for room classification of three-dimensional scene maps based on multimodal feature tensor decomposition according to claim 8, characterized in that, The 3D scene map contains room-level nodes with room type labels, and is used for high-level semantic navigation and task planning of intelligent robots.
10. A three-dimensional scene atlas room classification system based on multimodal feature tensor decomposition, characterized in that, include: The data acquisition module is configured to control the movement of a sensor-equipped intelligent agent in a target environment, and to collect data through the sensors. Observation data for each scene; Feature extraction module, configured to be used to extract features A different pre-trained model for the Feature extraction is performed on scene observation data, and the extracted feature vectors are organized according to the model. The set of characteristic matrices is the set of characteristic matrices. The tensor decomposition module is configured to perform tensor decomposition on the feature matrix set to simultaneously obtain: a common observation-class correlation matrix, which characterizes the weight of each observation belonging to each potential room class; And the feature-category factor matrix corresponding to each model, the column vector of the matrix represents the prototype features of the potential room category in the model feature space; where the rank of the tensor decomposition is determined as the number of potential room categories in the environment; The semantic mapping module is configured to fuse prototype features in each feature-category factor matrix for each potential room category to form a fused category prototype representation; by calculating the similarity between the category prototype representation and the semantic representation of the predefined room type label, the most matching room type label is assigned to each potential room category. The map update module is configured to determine the room category to which each scene observation belongs based on the observation-category association matrix, and assign the corresponding room type label to the room layer node in the 3D scene map, thereby adding semantic category information to the room layer.