Multi-modal large model space sensing method based on 3D scene graph driving

Abstract

The invention discloses a multi-modal large model space perception method based on 3D scene graph driving, and the method comprises the steps: taking a 3D semantic scene graph algorithm as an additional visual encoder of a multi-modal large model after obtaining multi-modal original information, extracting 3D object instance features in a scene and relation features between objects, and obtaining 3D perception capability; meanwhile, a structured scene topology description text is generated, and spatial perception representation with display and implicit coexistence is constructed; a point cloud punctuation mark mechanism based on geometric primitives is introduced, and standard geometry features are extracted and solidified through a shared 3D visual encoder to serve as anchor points for distinguishing different semantic units; a point cloud projection layer is constructed to uniformly map the features into semantic Token, and a composite input sequence is formed in combination with structured text prompt. Through a dual context guidance mechanism, depth space perception of a large model to a 3D scene is realized, and question and answer accuracy and logical reasoning ability of the model in a complex three-dimensional environment are significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision, specifically to a technique for constructing and optimizing a multimodal large language model (MLLM), particularly a technique for introducing a 3D semantic scene graph visual encoder to achieve multimodal perception and alignment of native 3D scenes. Background Technology

[0002] With the rapid development of deep learning technology, large language models (LLMs) have demonstrated powerful general reasoning capabilities. By introducing a visual encoder, multimodal large models (MLLMs) can simultaneously understand images and text, achieving significant progress in tasks such as visual question answering and image description. However, in applications such as embodied AI and robot navigation for the real physical world, existing multimodal large models still face serious challenges, mainly in the following aspects:

[0003] 1. Lack of native 3D spatial perception capabilities. Existing mainstream multimodal large models are mainly based on 2D image input, attempting to infer 3D spatial information through multi-view image projection. This dimensionality reduction method leads to the loss of depth information, blurred occlusion relationships, and inaccurate understanding of spatial topology (such as the relative positions and distances between objects), making it difficult for the model to construct a realistic and coherent 3D world model.

[0004] 2. Semantic confusion in long-sequence point cloud features. Although some studies attempt to directly incorporate 3D point cloud data, the raw point cloud data is usually converted into long and unordered token sequences. This method fails to construct the geometric features of object instances and the dependency features of relationships between objects, and the point cloud representations are mixed and lack clear semantic boundaries. This makes the model prone to "getting lost" in long-context reasoning.

[0005] 3. Lack of explicit spatial logic guidance. Purely data-driven training makes it difficult for models to capture complex spatial logic. Relying solely on implicit learning of point cloud features often leads to model illusions and makes it difficult to align with textual representations. There is a lack of structured prior knowledge to assist in large-scale model analysis and anchoring 3D representations.

[0006] Therefore, there is an urgent need for a new technical solution that can introduce native 3D scene graph representations into multimodal large models and solve the problem of large models' perception of 3D space through effective multimodal alignment and delimitation mechanisms. Summary of the Invention

[0007] To address the problems mentioned in the background section, this invention proposes a multimodal large-model spatial perception method driven by 3D scene graphs, employing the following technical solution:

[0008] A multimodal large model spatial perception method based on 3D scene graph-driven approach includes the following steps:

[0009] Step 1: Obtain the multimodal raw information of the scene to be analyzed, including scene images, user commands, and 3D point cloud data;

[0010] Step 2: Input the preprocessed point cloud data into the pre-trained 3D scene graph visual encoder to parse the object instances in the scene and the spatial topological relationships between objects, and extract the instance feature vector and the relationship feature vector respectively.

[0011] Step 3: Based on the parsing results, convert the 3D scene graph into structured JSON format text;

[0012] Step 4: Based on a predefined geometric primitive library, extract features from standard cube point clouds and standard sphere point clouds respectively, as instance punctuation features and relational punctuation features;

[0013] Step 5: Construct a point cloud projection layer to uniformly map the instance features, relation features, and punctuation features to the embedding space of the large language model;

[0014] Step 6: Sequentially assemble the mapped features, define the start and end symbols of the 3D spatial features, and form a structured point cloud modal input sequence;

[0015] Step 7: Concatenate the JSON formatted text, user text commands, 2D image information, and structured point cloud modal input sequence to construct a multimodal command input pool;

[0016] Step 8: Input the data from the multimodal instruction input pool into the large language model, and use the self-attention mechanism to generate a large model response that includes spatial semantic understanding.

[0017] Furthermore, step 1 includes the following steps:

[0018] The 3D point cloud scene data is preprocessed to remove sensor noise points, and the scene image and user commands are converted simultaneously. The scene image is used to extract features through a 2D visual encoder to obtain 2D scene image features; the user commands are converted into text embedding.

[0019] Furthermore, step 2 includes the following steps:

[0020] 3D point cloud scene after preprocessing of input The 3D scene graph perception algorithm is used as the point cloud visual encoder for the multimodal large model. The 3D scene graph representation, as well as the prediction results of scene graph nodes and edges, are obtained from the point cloud instance feature branch, the point cloud instance relationship extraction branch, and the prediction results of the relationship between point cloud instances.

[0021] Furthermore, step 3 includes the following steps:

[0022] Based on the prediction results generated in step 2, a mapping function is constructed. This process traverses all nodes and edges to generate JSON format text. The JSON format text contains the instance's category label, spatial bounding box attributes, and topological connection relationships, which are used to construct the prior context of the text.

[0023] Furthermore, step 4 includes the following steps:

[0024] Standard geometric primitives are defined as “punctuation” elements of 3D point cloud features. The geometric primitives include unit cube point clouds and unit sphere point clouds, which are used to represent instance punctuation and relational punctuation, respectively.

[0025] Using a 3D scene graph visual encoder Extracting punctuation features;

[0026] After extracting punctuation features, the 3D scene graph visual encoder freezes and solidifies them, and then directly calls the solidified feature vectors in subsequent inference steps.

[0027] Furthermore, the point cloud projection layer in step 5 is composed of a multi-layer perception mechanism, and its parameters are updated during the instruction fine-tuning stage of the multimodal large model to achieve alignment between the 3D geometric features of the scene and the text semantic space of the large model, while establishing a logical association between punctuation marks and subsequent semantic features.

[0028] The point cloud projection layer consists of a two-layer fully connected network. Through nonlinear mapping, the point cloud projection layer aligns the distribution of 3D geometric space to the text semantic space, enabling large models to process geometric representations in the same way as text tokens.

[0029] Furthermore, step 6 includes the following steps:

[0030] The mapped features are serialized and assembled to construct an input sequence with well-defined boundaries, and special control characters are defined.<point start> and<point end> The package sequence is assembled according to the "punctuation-content" pairing principle. A cube punctuation feature is added before each instance token, and a sphere punctuation feature is added before each relation token to form a structured 3D modal sequence.

[0031] Furthermore, step 7 includes the following steps:

[0032] The JSON-formatted text generated in step 3, the user text commands, the 2D scene image features processed in step 1, and the structured 3D modal sequence generated in step 6 are concatenated. All modal information is uniformly pushed into the input buffer of the large model to form a multimodal command pool containing pixels, geometric topology, and logical text.

[0033] Furthermore, step 8 includes the following steps:

[0034] The multimodal instruction pool data is input into the multimodal large language model decoder based on Qwen2.5-VL. During the training phase, the 3D scene graph visual encoder is kept frozen, and the parameters of the large model and the projection layer are fine-tuned using low-rank adaptive technology. The model generates accurate responses containing spatial semantic understanding through a self-attention mechanism, guided by the logic of JSON text and defined by the 3D information boundaries of "geometric punctuation".

[0035] Furthermore, during the training phase, the multimodal large model uses the next prediction task as the loss function, while simultaneously optimizing the spatial description accuracy in the text response and the semantic alignment of the point cloud modality.

[0036] The beneficial effects of this invention are as follows: This invention abandons the traditional approach of relying solely on 2D images to infer spatial information. By introducing a 3D semantic scene graph (3DSSG) as an intermediate representation, a 3D scene graph visual encoder is constructed, enabling large models to acquire the ability to natively perceive object instances and spatial topological relationships in 3D space. Simultaneously, through explicit injection of JSON-formatted text, a clear contextual index is provided to the model, improving the accuracy of spatial reasoning in complex scenes. Furthermore, addressing the pain point of semantic confusion in long point cloud sequences, this invention innovatively utilizes standard geometric elements to construct "point cloud punctuation marks," which play a clear separating and cuing role in 3D feature sequences, helping the attention mechanism to focus accurately. Attached Figure Description

[0037] Figure 1 This is an overview diagram of the present invention;

[0038] Figure 2 This is a schematic diagram of the method of the present invention;

[0039] Figure 3 This is a rendering of the invention. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and examples.

[0041] The overall architecture of the multimodal large model spatial perception method based on 3D scene graph-driven proposed in this invention is as follows: Figure 1 As shown in the diagram, the method example is illustrated below. Figure 2As shown.

[0042] Step 1: Acquisition and Preprocessing of Multimodal Raw Information

[0043] Acquire multimodal raw information of the scene to be analyzed, including scene images, user commands, and 3D point cloud data. For example... Figure 2 As shown in the top left corner, the scene point cloud is preprocessed to remove sensor noise points. Simultaneously, other multimodal information is transformed, such as scene image feature extraction using a 2D scene graph visual encoder, and user commands are converted into text embeddings.

[0044] For the input complete 3D point cloud scene Preprocessing is performed, in which For point cloud points, Represents XYZ coordinates, This represents RGB color information. Preferably, the point cloud preprocessing method is spatial location normalization and noisy point cloud filtering. Preferably, the filtering scheme uses statistical outlier filtering.

[0045] By calculating the distance from each point to its The average distance between the nearest neighbors And assume it follows a Gaussian distribution. Eliminate distances in the interval [missing information]. Outside of the points, among which The mean distance Standard deviation The threshold value is set to a multiple (preferably 1.0 to 3.0). This process can eliminate discrete noise generated when the sensor acquires point clouds, avoiding the misleading effect of isolated noise points on subsequent scene graph topology construction.

[0046] Step 2: 3D Semantic Scene Graph Feature Analysis

[0047] The preprocessed scene point cloud is input into a pre-trained 3D scene graph visual encoder, such as... Figure 2 As shown in the gray dashed box in the middle, the encoder based on the 3D scene graph has a dual-branch extraction capability: branch one extracts the feature vectors (instance features) of each object instance in the scene by stripping away object masks; branch two extracts spatial topological feature vectors (relationship features) based on the adjacency relationships between objects, thereby achieving a comprehensive analysis of the 3D space from the underlying geometry to the high-level semantics. Simultaneously, it obtains scene graph information prediction for the 3D space.

[0048] 3D point cloud scene after preprocessing of input A 3D scene graph perception algorithm is used as the point cloud visual encoder for a multimodal large model. The 3D scene graph representation, as well as the prediction results of scene graph nodes and edges, are obtained from the point cloud instance feature branch, the point cloud instance relationship extraction branch, and respectively.

[0049]

[0050] in For the scene The feature set of an object instance It is a feature set of spatial topological relationships between objects. This represents a pre-trained 3D scene graph visual encoder, preferably Granular3D. This represents the set of parameters for a pre-trained 3D scene graph encoder. Then it is the instance label predicted by the encoder. Relationship tags between instances The set of high-dimensional vectors obtained by introducing a scene graph encoder structure explicitly extracts the "entity" and "connection" features of the scene, transforming the disordered point cloud into graph features with semantic topological structure, and providing the model with spatial logical understanding beyond simple geometric shapes.

[0051] Step 3: Scene Graph Text Formatting Mapping

[0052] Based on the 3D scene graph structure generated in step 2, formatted mapping is performed using predefined rule templates. For example... Figure 2 As shown in the top right corner, the extracted object IDs, category labels, 3D bounding boxes, and predicate relationships between objects (such as "left", "near", "above") are converted into structured JSON text. This text serves as an explicit logical prior, enhancing the model's ability to reason about long-distance spatial relationships.

[0053] Based on the graph generated in step 2 Construct mapping function This process iterates through all nodes. With edge Generate JSON formatted text:

[0054]

[0055] This step involves "explicitly translating" the complex 3D spatial topology using natural language (JSON text), injecting it as strong prior knowledge into the context prompt of the large model, enabling the large model to quickly locate specific spatial entities in a long text context through logical indexing.

[0056] Step 4: Extraction and solidification of standard geometric punctuation features

[0057] To establish semantic boundaries in a continuous sequence of point cloud features, this invention introduces standard geometric primitives as "punctuation marks." For example... Figure 2As shown on the left, standard unit cube point clouds (representing instance punctuation) and standard unit sphere point clouds (representing relation punctuation) are used to extract their high-dimensional representations through a 3D scene graph visual encoder shared with the scene. The extracted punctuation features are then constantized and solidified to reduce redundant computation overhead.

[0058] This invention defines standard geometric primitives As a feature of 3D point clouds, the "punctuation mark" element, in which For the preferred unit cube point cloud, The preferred unit sphere point cloud is constructed using a uniformly distributed mesh, and the unit sphere is sampled using the furthest point at FPS to ensure that the point density inside the sphere is approximately the same in volume.

[0059] To ensure consistency in feature distribution, this step utilizes a 3D scene graph visual encoder that shares parameters with the backbone network from step 1. Extracting punctuation features:

[0060]

[0061] Meanwhile, in order to reduce the computational cost of punctuation features during training and inference, the preferred punctuation features are... and In multimodal large models, constants are solidified and dimensions are optimized and aligned to... This dimension feature is the same dimension as the point cloud instance features and inter-instance relationship features obtained from the 3D scene graph.

[0062] Introducing geometric punctuation marks can provide "semantic boundaries" for dense, continuous feature flows. Cube punctuation marks indicate that the model is followed by entity objects, while spherical punctuation marks indicate that the model is followed by relational descriptions, thus achieving a separation and alignment function similar to text punctuation in the latent space.

[0063] Step 5: Feature Mapping of Point Cloud Projector Layer

[0064] By utilizing a point cloud projection layer composed of a multilayer perceptron (MLP), the instance features and relational features extracted in step 2, as well as the cube and sphere punctuation features extracted in step 4, are uniformly mapped to the embedding space dimension of the large language model. For example... Figure 2 As shown in the blue and yellow blocks on the right, this step achieves dimensional and distributional alignment between the geometric representation and the text semantic space.

[0065] The PointCloud Projector Layer, which incorporates various 3D point cloud representations into the text space of a multimodal large model, consists of a two-layer fully connected network, and its mathematical expression is as follows:

[0066]

[0067] in, , dimension ; For the projection matrix, To map to the embedding space dimension of a large model The projection layer aligns the distribution of 3D geometric space to the text semantic space through nonlinear mapping, enabling large models to process geometric representations in the same way they process text tokens.

[0068] Step 6: Assembly of structured point cloud feature sequences

[0069] The mapped features are serialized and assembled to construct an input sequence with well-defined boundaries. This invention defines special control characters.<point start> and<point end> Package sequence. For example... Figure 2 As shown in the "Point Cloud Feature Assembly" step in the lower right corner, following the "punctuation-content" pairing assembly principle, a cube punctuation feature is added before each instance token, and a sphere punctuation feature is added before each relation token, forming a structured 3D modal sequence.

[0070] This invention defines special control characters<point start> and<point end> Constructing point cloud modal input sequences Assemble according to the paired structure of "punctuation mark - content":

[0071]

[0072] A cube punctuation token is appended to the instance feature sequence after the projection layer mapping. A sphere-shaped punctuation mark (Token) is appended between instance features and relation features. As a separator, this structured sequence mimics the organization of human language, not only explicitly marking the start and end of 3D data, but also enhancing the model's perception of object and relational boundaries through "geometric punctuation".

[0073] Step 7: Construction of a Unified Multimodal Instruction Pool

[0074] The scene graph text in JSON format generated in step 3, the user text commands, the 2D scene image features processed in step 1, and the structured point cloud feature sequence generated in step 6 are concatenated. For example... Figure 2 As shown at the bottom, all modal information is uniformly pushed into the input buffer of the large model, forming a multimodal instruction pool containing pixels, geometric topology, and logical text.

[0075] The multimodal large model based on 3D scene graph driven proposed in this invention unifies and combines system information, user commands, 2D image information, scene graph text information, and 3D point cloud scene graph spatial representation.

[0076]

[0077] in, This indicates that the text token is converted into text features through the embedding layer of a multimodal large model; , and These represent system information, user command information, and 2D image features, respectively. Through this multimodal collaborative learning, the model can simultaneously process information from three dimensions—2D pixels, 3D geometry, and scene graph-text logic—within the same self-attention space, thereby improving its scene perception capabilities.

[0078] Step 8: Large Model Self-Attention Decoding and Inference Generation

[0079] Multimodal instruction pool data is input into a Qwen2.5-VL-based MLLM Decoder. During training, the 3D encoder is kept frozen, and LoRA (Low-Rank Adaptive) technique is used to fine-tune the large model parameters and projection layers. Through a self-attention mechanism, guided by the logic of the JSON text and defined by the 3D information boundaries of "geometric punctuation," the model generates accurate responses with spatial semantic understanding.

[0080] The multimodal large model utilizes the data obtained in step 7. The mixed sequences are processed using a multi-head self-attention mechanism to generate the final result. Preferably, the multimodal large model backbone used in this invention is the Qwen3-VL series multimodal large model. Preferably, this invention uses the LoRA mechanism to fine-tune the weights of the large model, and its training parameters... Represented as:

[0081]

[0082] in It should be a low-rank number, preferably 8. This represents the token dimension of the model. Simultaneously, during the LoRA process, the model trains the point cloud projection layer (Projector) and freezes the parameters of the 3D scene graph visual encoder (Granular3D). Preferably, the model involved in this invention is optimized using an auto-regressive loss function:

[0083]

[0084] in, and These are the LoRA parameters and 3D projection layer parameters of the model, respectively. By freezing the pre-trained 3D encoder and fine-tuning the LoRA and projection layers, the powerful spatial representation capabilities of Granular3D are preserved, while achieving deep alignment between 3D geometric logic and the powerful language understanding capabilities of Qwen2.5-VL with extremely low computational cost.

[0085] like Figure 3 The diagram shows the operational effect of the method of this invention. The two examples in the first row demonstrate that the multimodal large model constructed by this scheme has the ability to locate user-described objects in the scene, and can respond with accurate height and volume estimates for specific objects based on user instructions and the precise encoding of spatial information by the 3D point cloud modality. The examples on the left in the second row show the ability of the multimodal large model proposed in this invention to count objects in the scene; the examples on the right show the model's ability to locate multiple objects and estimate spatial distances. These two examples further highlight the excellent performance of the algorithm in the overall scene perception and reasoning task. Experimental results show that this method performs outstandingly in 3D spatial understanding and object relationship reasoning tasks. Through native 3D scene graph feature perception, combined with the display scene graph JSON expression, the model's grounding ability and the accuracy of spatial parameter prediction are significantly improved compared to traditional image-text multimodal models.

[0086] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multimodal large-model spatial perception method based on 3D scene graph-driven approach, characterized in that, Includes the following steps: Step 1: Obtain the multimodal raw information of the scene to be analyzed, including scene images, user commands, and 3D point cloud data; Step 2: Input the preprocessed point cloud data into the pre-trained 3D scene graph visual encoder to parse the object instances in the scene and the spatial topological relationships between objects, and extract the instance feature vector and the relationship feature vector respectively. Step 3: Based on the parsing results, convert the 3D scene graph into structured JSON format text; Step 4: Based on a predefined geometric primitive library, extract features from standard cube point clouds and standard sphere point clouds respectively, as instance punctuation features and relational punctuation features; Step 5: Construct a point cloud projection layer to uniformly map the instance features, relation features, and punctuation features to the embedding space of the large language model; Step 6: Sequentially assemble the mapped features, define the start and end symbols of the 3D spatial features, and form a structured point cloud modal input sequence; Step 7: Concatenate the JSON formatted text, user text commands, 2D image information, and structured point cloud modal input sequence to construct a multimodal command input pool; Step 8: Input the data from the multimodal instruction input pool into the large language model, and use the self-attention mechanism to generate a large model response that includes spatial semantic understanding.

2. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 1 includes the following steps: The 3D point cloud scene data is preprocessed to remove sensor noise points, and the scene image and user commands are converted simultaneously. The scene image is used to extract features through a 2D visual encoder to obtain 2D scene image features; the user commands are converted into text embedding.

3. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 2 includes the following steps: For the preprocessed 3D point cloud scene input, a 3D scene graph perception algorithm is used as the point cloud visual encoder for a multimodal large model. The 3D scene graph representation, as well as the prediction results of scene graph nodes and edges, are obtained from the point cloud instance feature branch, the point cloud instance relationship extraction branch, and the prediction results of the relationship between point cloud instances.

4. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 3 includes the following steps: Based on the prediction results generated in step 2, a mapping function is constructed. This process traverses all nodes and edges to generate JSON format text. The JSON format text contains the instance's category label, spatial bounding box attributes, and topological connection relationships, which are used to construct the prior context of the text.

5. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 4 includes the following steps: Standard geometric primitives are defined as "punctuation mark" elements of 3D point cloud features. The geometric primitives include unit cube point clouds and unit sphere point clouds, which are used to represent instance punctuation and relational punctuation, respectively. Extracting punctuation features using a 3D scene graph visual encoder; After extracting punctuation features, the 3D scene graph visual encoder freezes and solidifies them, and then directly calls the solidified feature vectors in subsequent inference steps.

6. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, The point cloud projection layer in step 5 is composed of a multi-layer perception mechanism. Its parameters are updated during the instruction fine-tuning stage of the multimodal large model to achieve alignment between the scene's 3D geometric features and the text semantic space of the large model, while establishing a logical association between punctuation marks and subsequent semantic features. The point cloud projection layer consists of a two-layer fully connected network. Through nonlinear mapping, the point cloud projection layer aligns the distribution of 3D geometric space to the text semantic space, enabling large models to process geometric representations in the same way as text tokens.

7. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 6 includes the following steps: The mapped features are serialized and assembled to construct an input sequence with well-defined boundaries, and special control characters are defined.<point start> and<point end> The package sequence is assembled according to the "punctuation-content" pairing principle. A cube punctuation feature is added before each instance token, and a sphere punctuation feature is added before each relation token to form a structured 3D modal sequence.

8. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 7 includes the following steps: The JSON-formatted text generated in step 3, the user text commands, the 2D scene image features processed in step 1, and the structured 3D modal sequence generated in step 6 are concatenated. All modal information is uniformly pushed into the input buffer of the large model to form a multimodal command pool containing pixels, geometric topology, and logical text.

9. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, Step 8 includes the following steps: The multimodal instruction pool data is input into the multimodal large language model decoder based on Qwen2.5-VL. During the training phase, the 3D scene graph visual encoder is kept frozen, and the parameters of the large model and the projection layer are fine-tuned using low-rank adaptive technology. The model generates accurate responses containing spatial semantic understanding through a self-attention mechanism, guided by the logic of JSON text and defined by the 3D information boundaries of "geometric punctuation".

10. The multimodal large model spatial perception method based on 3D scene graph driven according to claim 1, characterized in that, During the training phase, the multimodal large model uses the next prediction task as the loss function, while optimizing the spatial description accuracy in the text response and the semantic alignment of the point cloud modality.

Images