A three-dimensional scene understanding and instruction analysis method based on a multi-modal large model

Abstract

The application belongs to the technical field of deep learning, and discloses a three-dimensional scene understanding and instruction analysis method based on a multi-modal large model. The method first collects and pre-processes multi-modal data around the target application scene, completes space-time alignment and semantic annotation, and constructs a standardized data set. Then a multi-modal scene understanding large model is constructed, each modal feature is extracted through a special encoder, multi-modal fusion and dynamic feature enhancement are realized by using deformable attention and time series modeling, and a sparse mixed expert architecture is introduced to improve the model expression ability and efficiency. In the training stage, a phased strategy is adopted, first aligning multi-modal semantics, then training modal experts, finally integrating them into the large model for joint fine-tuning, and using GOAT technology to realize lightweight parameter fine-tuning. Finally, the model is deployed in the actual environment to realize automatic perception and intelligent decision-making in complex scenes. The application significantly improves the precision and efficiency of three-dimensional scene understanding, and is suitable for fields such as intelligent manufacturing and autonomous driving.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and particularly relates to a three-dimensional scene understanding and instruction analysis method based on a multi-modal large model. BACKGROUND

[0002] With the rapid development of deep learning and large-scale neural network technology in recent years, multi-modal large models have made significant progress in the fields of vision, language, etc. Large models have shown strong cross-modal understanding and reasoning capabilities, and can fuse images, texts, audios, etc. to achieve cross-modal semantic alignment and joint reasoning, and have great potential in intelligent interaction, autonomous driving, virtual reality, etc.

[0003] Information in the real world is naturally diverse and complex, and is usually presented in the form of coexistence of multiple modalities such as images, texts, audios, videos, and point clouds. The modeling method of a single modality has been difficult to meet the demand for comprehensive perception and deep understanding of complex scenes. Existing technologies still face many challenges in dealing with complex real-world scenes. The current mainstream multi-modal model mainly focuses on the joint processing of images, videos, and texts, and the support for other important modalities such as 3D point clouds, sketches, and audios is relatively weak, which is difficult to meet the demand for collaborative analysis of multi-source heterogeneous data in industrial detection, robot interaction, etc. Especially in the aspect of three-dimensional scene understanding, existing methods often rely on 2D projection, resulting in loss of geometric details, and perform poorly in dynamic scene analysis tasks.

[0004] Existing methods still have obvious deficiencies in multi-modal unified representation, three-dimensional semantic decoupling, and dynamic sparse calculation, etc. Especially in dealing with complex scenes, there is a lack of effective modeling capability for hierarchical semantic structure, making it difficult to achieve fine-grained scene understanding and generation. These technical bottlenecks seriously restrict the deep application of multi-modal large models in intelligent manufacturing, smart cities, etc. SUMMARY

[0005] The application aims to provide a three-dimensional scene understanding and instruction analysis method based on a multi-modal large model to solve the above technical problems.

[0006] To solve the above technical problems, the specific technical solutions of the three-dimensional scene understanding and instruction analysis method based on a multi-modal large model of the application are as follows:

[0007] A three-dimensional scene understanding and instruction analysis method based on a multi-modal large model, comprising the following steps:

[0008] Step 1: Collect multi-modal raw data around the application scene, complete preprocessing, spatio-temporal alignment, semantic annotation and structured organization, and build a standardized multi-modal scene understanding dataset available;

[0009] Step 2: Build a multi-modal scene understanding large model, sequentially complete modal feature extraction, semantic space alignment, temporal relationship modeling and expert fusion, and introduce a GOAT module at the key layer to realize efficient parameter fine-tuning;

[0010] Step 3: Based on the multi-modal scene understanding dataset, train the multi-modal scene understanding large model;

[0011] Step 4: After training and evaluation, deploy the multi-modal scene understanding large model in the target application environment to complete automatic perception and intelligent decision-making in real or quasi-real-time scenarios.

[0012] Further, the step 1 includes the following steps:

[0013] Step 1.1: Multi-modal data acquisition: Collect multi-modal data with representativeness and diversity from real scenes or open platforms around target application scenarios, covering visual, three-dimensional space, auditory, graphical and textual modalities;

[0014] Step 1.2: Multi-modal data preprocessing and alignment: Perform unified preprocessing and alignment on the data of different modalities collected in step 1.1;

[0015] Step 1.3: Semantic annotation and structured organization: Based on data alignment, introduce multi-level semantic annotation, build label structure required for supervised learning, and finally organize each modality data and label in a unified format to form a structured sample set, providing a standard interface for model input and training;

[0016] Step 1.4: Data format construction and dataset division: Encapsulate the processed multi-modal samples into a unified data format and divide them into training set, validation set and test set.

[0017] Further, the step 2 includes the following steps:

[0018] Step 2.1: For data from different modalities, respectively adopt exclusive strategies for feature encoding;

[0019] Step 2.2: Temporal modeling and dynamic feature enhancement: For data with time continuity, introduce position encoding mechanism to embed time information into modality representation; then input the feature with position encoding into the temporal modeling module based on Transformer structure, use its self-attention mechanism to capture the temporal dependence relationship within the modality, model scene changes, action continuity and dynamic behavior patterns;

[0020] Step 2.3: Query vector driven semantic focus modeling: Through cross-attention calculation with input modality features, extract structured semantic content, generate modality-level high-level semantic representation highly aligned with language instructions;

[0021] Step 2.4: Sparse expert dynamic activation and feature fusion: Based on the aligned multi-modal semantic representation, a sparse mixed expert architecture is introduced to enhance the adaptability and expression ability of the model;

[0022] Step 2.5: Lightweight parameter fine-tuning mechanism: Introduce GOAT mechanism, the model uses GOAT technology for lightweight parameter fine-tuning.

[0023] Further, the step 2.1 includes the following steps:

[0024] For different modal data from images, point clouds, audio, video, sketches and text, respectively, adopt exclusive strategy for feature encoding, video through average pooling aggregation multi-frame features and through CLIP encoder for feature extraction, sketch through CLIP encoder for feature extraction, audio adopts Whisper encoder, point cloud adopts I2P-MAE as 3D encoder, text adopts Bert model as text encoder; Point cloud performs dynamic voxelization and sparse 3D convolution to extract geometric features from point cloud, and finally compresses into BEV space feature representation; Multi-view RGB image extracts multi-view image features through CNN and combines with depth prediction, generates semantic rich BEV feature map through 3D projection and voxel pooling; Then enter the feature fusion stage, the dynamic feature fusion layer adaptively integrates the geometric information of laser radar and the semantic information of camera through deformable attention mechanism, and the time series enhancement module aligns and fuses the historical features to the current frame through motion compensation and cross-frame attention, forming a spatio-temporal consistent feature representation, and finally outputting unified BEV feature.

[0025] Further, the step 2.4 performs sparse activation on the multi-modal expert network through the gating mechanism, and only selects the top 2 related experts for execution in each input, and dynamically aggregates the output results.

[0026] Further, the step 2.5 only inserts low-rank weight transformation to part of the key layers, and freezes the original parameters, only optimizes the GOAT substructure, the core idea of GOAT is to model the output increment of the expert as a linear combination of several low-rank subspaces, where each subspace is defined by a dynamic SVD structure.

[0027] Further, the step 3 includes the following steps:

[0028] Step 3.1: Align multi-modal feature semantics and build unified language representation space;

[0029] Step 3.2: Train modal experts to improve single-modal understanding ability;

[0030] Step 3.3: Integrate modal experts and fine-tune sparse hybrid architecture.

[0031] Further, step 3.1 extracts structured semantic features from images, videos, audio, point clouds, sketches, and text by constructing a spatiotemporal alignment framework, and maps them to a unified language semantic space; image and depth map inputs are enhanced by a 3D structure perception enhancement module for better spatial understanding; video frames are averaged and then encoded by a CLIP encoder to extract semantics; sketches are also encoded by CLIP; audio is encoded by a Whisper encoder, and text is encoded by BERT; each modal feature is mapped to the language model input domain through a lightweight connector.

[0032] Further, step 3.2 optimizes the feedforward neural network structure by minimizing the generative entropy loss between modal features and semantic labels, making it better adapt to modal feature expression.

[0033] Further, step 3.3 integrates the trained modal experts into the sparse hybrid expert architecture of the LLM model, which consists of shared attention modules, multiple experts, and a dynamic router. The router dynamically activates Top-k experts and weights their outputs based on input tokens, enabling context-driven expert selection. The expert weight distribution mechanism uses a Top-k Softmax routing strategy. A load balancing loss term is introduced to encourage balanced utilization of experts. Various cross-modal training samples are introduced for joint fine-tuning, with only the expert modules updated during fine-tuning. GOAT technology is used to reduce parameter overhead.

[0034] The three-dimensional scene understanding and instruction analysis method based on a multi-modal large model has the following advantages:

[0035] 1. Comprehensive improvement of multi-modal scene understanding capability

[0036] By fusing image, point cloud, audio, video, sketch, and text modal data, a unified multi-modal semantic representation space is constructed, significantly enhancing the perception and understanding of complex three-dimensional scenes.

[0037] Introducing temporal modeling and dynamic feature enhancement mechanisms effectively captures scene changes, behavior continuity, and dynamic patterns, suitable for real-time analysis and decision-making in dynamic environments.

[0038] 2. Efficient multi-modal fusion and alignment mechanism

[0039] Adopting deformable attention mechanism (MDA) and temporal augmentation module (TAM), adaptive fusion of laser radar and camera information is realized, generating spatiotemporally consistent BEV features, preserving geometric structure and integrating semantic context.

[0040] Cross-modal semantic alignment is achieved through Q-Former + linear projection, mapping multi-modal features to a unified language space, improving the model's understanding and response ability to language instructions.

[0041] 3. Lightweight and efficient model structure and training strategy

[0042] Introduce sparse mixture-of-experts (MoE) architecture, only activate Top-2 experts for inference, significantly reduce computational overhead, improve model efficiency and scalability.

[0043] Use GOAT lightweight fine-tuning technology, only optimize low-rank adapters, freeze the backbone parameters, achieve fast task adaptation and deployment, significantly reduce the parameter quantity and training cost of fine-tuning.

[0044] 4. Strong generalization ability and task adaptability

[0045] Support multiple downstream tasks such as object detection, behavior recognition, instruction parsing, anomaly detection, etc., suitable for intelligent manufacturing, autonomous driving, robot interaction, smart city and other fields.

[0046] Through load balancing loss and diversified training samples, improve expert utilization and model stability, avoid expert idling or overfitting.

[0047] 5. End-to-end intelligent decision support

[0048] The model can directly receive multi-source sensor input and natural language instructions, output structured semantic results, drive downstream systems (such as robots, AR systems, industrial control systems) for automatic execution, forming a "perception-understanding-decision-execution" closed loop.

[0049] Excellent performance in real-time or quasi-real-time scenarios, high precision, high robustness and strong practicality.

[0050] 6. Standardization and scalability

[0051] Provide complete processes from data collection, preprocessing, labeling to model training, deployment, support standardized interfaces and modular design, facilitate subsequent function extension and system integration. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is the system module block diagram of the present invention.

[0053] Figure 2 is a dynamic variable fusion perception module flowchart in an embodiment of the present invention.

[0054] Figure 3 is a training flowchart in an embodiment of the present invention.

[0055] Figure 4is various case example schematic diagram of model generation in the embodiment of the application. DETAILED DESCRIPTION

[0056] In order to better understand the purpose, structure and function of the present application, the following will be combined with the drawings to further describe the three-dimensional scene understanding and instruction analysis method based on a multi-modal large model in detail.

[0057] As shown in Figure 1 The three-dimensional scene understanding and instruction analysis method based on a multi-modal large model of the present application comprises the following steps:

[0058] Step 1: Collect multi-modal raw data around the application scene, complete preprocessing, space-time alignment, semantic annotation and structured organization, and build a standardized and available multi-modal scene understanding dataset.

[0059] Step 1.1: Multi-modal data collection: Collect multi-modal data with representativeness and diversity from real scenes or open platforms around the target application scene, covering visual (such as multi-view RGB image, video, depth map), three-dimensional space (such as point cloud, structured light depth map), auditory (such as environmental sound, voice interaction), graphical (such as sketch, drawing) and textual (such as description sentence, operation instruction) modalities. Data sources include real sampling, simulation generation or public dataset, ensuring data diversity and semantic complexity, providing a comprehensive and reliable perception basis for subsequent model training.

[0060] Step 1.2: Multi-modal data preprocessing and alignment: Uniform preprocessing and accurate alignment of different modal data collected in step 1.1 to improve its quality and structural consistency. Images and videos need to be standardized in size and frame rate, audio needs to be denoised and sampled uniformly, and point cloud needs to be normalized in coordinates and regularized in point number. Each modality needs to be synchronized and aligned according to timestamp, location, etc. to ensure that the modal data corresponds to each other in the same scene. For some missing modal information, interpolation or generation techniques can be introduced to complete it, improving the availability and integrity of the overall data.

[0061] Step 1.3: Semantic annotation and structured organization: Based on data alignment, introduce multi-level semantic annotation to build the label structure required for supervised learning. Including annotation of object categories, positions and attributes, semantic description of events and actions, and generation of language instructions and task targets. The annotation process combines automatic tools and manual verification to ensure accuracy and efficiency. Finally, each modality data and label is organized in a unified format to form a structured sample set, providing a standard interface for model input and training.

[0062] Step 1.4: Data format construction and dataset division: The processed multi-modal samples are packaged into a unified data format and divided into training, validation, and test sets to ensure the scientificity of model training and evaluation. Each sample record contains fields such as modality data path, semantic label, and modality time information.

[0063] Step 2: Build a multi-modal scene understanding large model, sequentially complete modality feature extraction, semantic space alignment, temporal relationship modeling, and expert fusion, and introduce a GOAT module at the key layer to achieve efficient parameter fine-tuning. The overall architecture builds hierarchical semantic representations through dedicated encoders, cross-attention mechanisms, position encoding, and sparse expert routers, ultimately forming a unified multi-modal representation model with strong generalization ability and task adaptability.

[0064] Step 2.1: For different modality data such as images, point clouds, audio, videos, sketches, and text, respectively adopt exclusive strategies for feature encoding. Video aggregates multi-frame features through average pooling and performs feature extraction through a CLIP encoder, sketch performs feature extraction through a CLIP encoder, audio uses a Whisper encoder, point cloud uses I2P-MAE as a 3D encoder, and text uses a Bert model as a text encoder. As shown in Figure 2 , in the dynamic variable fusion perception module, point cloud is dynamically voxelized and sparse 3D convolution is used to extract geometric features from point cloud, and finally compressed into feature representation in BEV space; multi-view RGB images are extracted through CNN to extract multi-view image features and combined with depth prediction, 3D projection and voxel pooling to generate semantic-rich BEV feature map. Subsequently, in the feature fusion stage, the dynamic feature fusion layer adaptively integrates the geometric information of LiDAR and the semantic information of Camera through the deformable attention mechanism (MDA), and the temporal augmentation module (TAM) aligns and fuses the historical features to the current frame using motion compensation and cross-frame attention, forming a spatio-temporal consistent feature representation. The final output unified BEV feature not only retains the accurate spatial structure, but also integrates rich visual semantics and temporal context, which can be seamlessly connected to various downstream task heads through a standardized interface. The core formula is as follows:

[0065]

[0066] wherein represents the BEV feature generated by LiDAR point cloud, represents the input original LiDAR point cloud data, represents the conversion of unordered point cloud to regular voxel grid, represents the extraction of voxel features by sparse 3D convolution network, represents the compression along the height dimension to generate the final BEV feature map. BEV feature generated by camera image, input multi-view image, convolutional neural network for extracting image feature, D predicted depth distribution, camera intrinsic and extrinsic parameters, operation of projecting 2D image feature to 3D space, operation of aggregating 3D feature to BEV plane, feature map size after LiDAR point cloud is voxelized and processed by sparse 3D convolutional network, where number of feature channels, reflecting semantic dimension extracted by network, spatial resolution after downsampling in Y direction with step size s, spatial resolution after downsampling in X direction with step size s, fused multi-modal BEV feature, BEV query feature generated by preliminary fusion of multi-view RGB image and point cloud, and respectively represent deformable attention calculation on point cloud and multi-view RGB image feature, projection matrix of the th head in multi-head attention. number of projection matrices.

[0067] All modal features are input into a dedicated connector composed of Q-Former and linear projection layer to realize cross-modal semantic alignment. The Q-Former module extracts language-related semantic representation through cross-attention mechanism, and then performs linear projection to finally map to unified language space. The calculation formula of this process is:

[0068]

[0069] wherein, is a semantic extraction module of modal feature, outputting a fixed-length token sequence; and are weight and bias items of linear mapping layer respectively; the final is the aligned language space semantic representation.

[0070] Step 2.2: Temporal modeling and dynamic feature enhancement: For data with temporal continuity (such as video frame sequences, continuous audio, point cloud trajectories, etc.), a position encoding mechanism is introduced to embed temporal information into the modal representation. Then the feature with position encoding is input into the temporal modeling module based on the Transformer structure, which uses its self-attention mechanism to capture the temporal dependence within the modal, effectively modeling scene changes, action continuity, and dynamic behavior patterns, and improving the expression ability of complex spatio-temporal information.

[0071] Step 2.3: Query vector driven semantic focus modeling: In the Q-Former module, a set of learnable query vectors is introduced for each modality, representing the "attention instructions" for the key semantics of that modality. Through cross-attention calculation with input modal features, query vectors can actively focus on the most relevant information area, extract structured semantic content, and generate modal-level high-level semantic representations highly aligned with language instructions. This query mechanism significantly enhances the model's cross-modal alignment ability and interpretability.

[0072] Step 2.4: Sparse expert dynamic activation and feature fusion: Based on the aligned multi-modal semantic representation, a sparse mixed expert architecture is further introduced to enhance the adaptability and expression ability of the model. This module uses a gating mechanism to activate the multi-modal expert network sparsely, selecting only the top 2 relevant experts for execution in each input, and dynamically aggregating their output results. This strategy not only improves the model's generalization ability for different task scenarios, but also significantly reduces computational overhead, making the overall inference process more efficient and flexible.

[0073] Step 2.5: Lightweight parameter fine-tuning mechanism: To improve the model's rapid adaptation ability in new scenarios or new tasks, the GOAT mechanism is introduced, and the model uses GOAT technology for lightweight parameter fine-tuning. Only low-rank weight transformation is inserted into part of the key layers, and the original parameters are frozen, and only the GOAT substructure is optimized. The core formula of its expert fusion output is as follows:

[0074]

[0075] where, is the input vector, is the original full connection weight matrix, where, represents the original output of the expert without any fine-tuning, represents the input vector sequence input to the expert, represents the forward calculation function of the first expert. represents the incremental output generated by the GOAT path, represents the output of the expert New module computation function after integrating GOAT adapter. The core idea of GOAT is to model the output increment of an expert as a linear combination of several low-rank subspaces, where each subspace is defined by a dynamic SVD structure. This mechanism significantly reduces the number of fine-tuning parameters while preserving the model's expressive power and stability.

[0076] Step 3: As shown in Figure 3 , based on the multi-modal scene understanding dataset, the multi-modal scene understanding large model is trained.

[0077] Step 3.1: Align multi-modal feature semantics and construct a unified language representation space: Through the construction of a spatio-temporal alignment framework for multi-modal data, structured semantic features are extracted from images, videos, audio, point clouds, sketches, and text, and mapped to a unified language semantic space. Image and depth map inputs are enhanced by a 3D structure perception enhancement module to enhance spatial understanding; video frames are averaged and then encoded by a CLIP encoder to extract semantics; sketches are also encoded by CLIP; audio uses a Whisper encoder, and text is encoded by BERT. Each modality feature is mapped to the language model input domain through a lightweight connector, such as the audio modality feature after being extracted by the Audio-Q Former and connected to the LLM model. In this stage, only the connector parameters are trained, and the language model and modality encoder are frozen. Semantic alignment is achieved through generative loss, which improves the language model's understanding of multi-modal input.

[0078] Step 3.2: Train modality experts to improve single-modality understanding: After completing multi-modal semantic alignment, train each modality's expert network to improve its modeling ability for modality features. By minimizing the generative entropy loss between modality features and semantic labels, optimize the feedforward neural network structure to better adapt to modality feature expression. For example, in audio expert training, only the FFN layer is optimized, and the rest remains frozen. This approach enhances the expert's ability to perceive modality details and provides a strong generalization modality basis for the subsequent sparse mixing expert system.

[0079] Step 3.3: Integrate modality experts and fine-tune sparse mixing structure: Integrate the trained modality experts into the sparse mixing expert architecture of the LLM model, which consists of shared attention modules, multiple experts, and a dynamic router. The router dynamically activates Top-k experts and weights their outputs based on input tokens, achieving context-driven expert selection. The expert weight distribution mechanism uses a Top-k Softmax routing strategy:

[0080]

[0081] where, is the router's activation of the first expert's score, Top-k expert set with the highest score in the current input, weight assigned to an expert, Softmax amplifies the relative weight of high-scoring experts and suppresses irrelevant experts from participating.

[0082] To solve the problem of frequent activation of some experts and long-term idling of the rest, a load balancing loss term is introduced to encourage balanced utilization of experts. The loss is defined as follows:

[0083]

[0084] where, the number of samples in the current batch, the total number of experts, denotes the sample the assigned weight of the th expert, the average usage rate of the th expert in the batch. When all experts are used equally (i.e. ), the loss reaches a minimum value of 1, and when some experts are monopolized for a long time, the value increases significantly. By minimizing this loss, the model capacity utilization efficiency and training stability can be effectively improved. In this phase, a variety of cross-modal training samples (such as video-audio-text triplets, point cloud-text pairs, sketch-text pairs, multi-view RGB+depth-text triplets) are introduced for joint fine-tuning. Only the expert modules are updated in fine-tuning, and the parameter overhead is reduced through the GOAT technique.

[0085] Step 4: After training and evaluation are completed, the multi-modal scene understanding large model is deployed in the target application environment to complete automatic perception and intelligent decision-making of complex tasks in real or quasi-real-time scenarios.

[0086] As shown in

[0087] , by integrating the trained model into actual business processes, the system can receive heterogeneous perception inputs from multiple sources (such as images, audio, point clouds, text, etc.) and combine them with user natural language instructions for joint inference to complete downstream tasks such as scene semantic understanding, target recognition and positioning, behavior recognition, anomaly detection, task instruction analysis, interactive question and answer, etc. Figure 4

[0088] ​For example, for the instruction: "Please review the personnel who stayed in the warehouse area for more than 5 minutes from 10:00 to 11:00 this morning", the system can output a structured response: "Personnel B is detected, entered the warehouse area at 10:23, stayed for 8 minutes and 42 seconds, carrying a yellow backpack", and further trigger the corresponding risk prompt or record archive.

[0089] In addition, the model inference result can also be used as high-level semantic input to drive downstream intelligent systems for automatic execution, such as guiding mobile robot path planning, assisting augmented reality system scene rendering, or linking industrial control systems to achieve precise response, building an intelligent closed loop from "multimodal understanding" to "scene execution", and comprehensively improving the environmental perception ability and task processing efficiency of the system.

[0090] It can be understood that the present application is described by some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A multi-modal large model-based three-dimensional scene understanding and instruction analysis method, characterized in that, Comprising the following steps: Step 1: Collect multi-modal raw data around application scenarios, complete preprocessing, spatio-temporal alignment, semantic annotation and structured organization, and construct a standardized multi-modal scene understanding dataset; Step 2: Construct a multi-modal scene understanding large model, sequentially complete modal feature extraction, semantic space alignment, temporal relationship modeling and expert fusion, and introduce a GOAT module at the key layer to realize efficient parameter fine-tuning; Step 2.1: For data from different modalities, respectively adopt exclusive strategies for feature encoding; Step 2.2: Temporal modeling and dynamic feature enhancement: For data with temporal continuity, introduce a position encoding mechanism to embed temporal information into modal representation; then input the feature with position encoding into a temporal modeling module based on the Transformer structure, use its self-attention mechanism to capture the temporal dependence relationship within the modal, model the scene change, action continuity and dynamic behavior pattern; Step 2.3: Query vector driven semantic focus modeling: Through cross-attention calculation with input modal features, extract structured semantic content, and generate modal-level high-level semantic representation highly aligned with language instructions; Step 2.4: Sparse expert dynamic activation and feature fusion: Based on the aligned multi-modal semantic representation, introduce a sparse mixed expert architecture to enhance the adaptability and expression ability of the model; Step 2.5: Lightweight parameter fine-tuning mechanism: Introduce the GOAT mechanism, and the model uses GOAT technology for lightweight parameter fine-tuning; Step 3: Based on the multi-modal scene understanding dataset, train the multi-modal scene understanding large model; Step 4: After completing training and evaluation, deploy the multi-modal scene understanding large model in the target application environment, and complete automatic perception and intelligent decision-making in real or quasi-real-time scenarios.

2. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 1, characterized in that, The step 1 comprises the following steps: Step 1.1: Multi-modal data acquisition: Collect multi-modal data with representativeness and diversity from real scenes or open platforms around target application scenarios, covering visual, three-dimensional space, auditory, graphical and text modalities; Step 1.2: Multi-modal data preprocessing and alignment: Perform unified preprocessing and alignment on the data collected in step 1.1 from different modalities; Step 1.3: Semantic annotation and structured organization: Based on data alignment, introduce multi-level semantic annotation, construct the label structure required for supervised learning, and finally organize each modality data and label according to a unified format to form a structured sample set, providing a standard interface for model input and training; Step 1.4: Data format construction and dataset division: Encapsulate the processed multi-modal samples into a unified data format, and divide them into training set, validation set and test set.

3. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 1, characterized in that, The step 2.1 comprises the following steps: Different modal data from images, point clouds, audio, videos, sketches and texts are encoded by dedicated strategies. Videos are aggregated by average pooling and extracted by a CLIP encoder. Sketches are extracted by a CLIP encoder. Audio is encoded by a Whisper encoder. Point clouds are extracted by I2P-MAE as a 3D encoder. Texts are encoded by a Bert model. Point clouds are dynamically voxelized and sparsely 3D-convoluted to extract geometric features from the point clouds and finally compressed into a BEV space feature representation. Multi-view RGB images are extracted by CNNs to extract multi-view image features and combined with depth prediction to generate a semantic-rich BEV feature map through 3D projection and voxel pooling. Then, the features are fused in a dynamic feature fusion layer that adaptively integrates the geometric information of the laser radar and the semantic information of the camera through a deformable attention mechanism. Meanwhile, a temporal enhancement module aligns and fuses historical features to the current frame using motion compensation and cross-frame attention to form a spatio-temporally consistent feature representation. Finally, a unified BEV feature is output.

4. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 1, characterized in that, The step 2.4 performs sparse activation on the multi-modal expert network through a gating mechanism, selects only the top 2 related experts in each input to execute, and dynamically aggregates the output results.

5. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 1, characterized in that, The step 2.5 only inserts a low-rank weight transformation for part of the key layers and freezes the original parameters, only optimizing the GOAT substructure. The core idea of GOAT is to model the output increment of the expert as a linear combination of several low-rank subspaces, where each subspace is defined by a dynamic SVD structure.

6. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 1, characterized in that, The step 3 includes the following steps: Step 3.1: Align multi-modal feature semantics and construct a unified language representation space; Step 3.2: Train modal experts to improve single-modal understanding ability; Step 3.3: Fuse modal experts and fine-tune the sparse mixing structure.

7. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 6, characterized in that, The step 3.1 constructs a spatio-temporal alignment framework for multi-modal data to extract structured semantic features from images, videos, audio, point clouds, sketches and texts and map them to a unified language semantic space. Images and depth maps are input into a 3D structure perception enhancement module to enhance spatial understanding. Video frames are extracted by a CLIP encoder after average pooling. Sketches are also encoded by a CLIP encoder; Audio is encoded by a Whisper encoder, and text is encoded by a BERT. Each modal feature is mapped to the language model input domain through a lightweight connector.

8. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 6, characterized in that, The step 3.2 optimizes the feedforward neural network structure by minimizing the generative entropy loss between modal features and semantic labels, making it better adapt to modal feature expression.

9. The multi-modal large model based three-dimensional scene understanding and instruction analysis method according to claim 6, characterized in that, The step 3.3 integrates the trained modal experts into a sparse mixing expert architecture of the LLM model, which is composed of a shared attention module and multiple experts, a dynamic router, the router dynamically activates Top-k experts according to input tokens and weights and fuses the outputs of the experts, realizes context-driven expert selection, and the expert weight distribution mechanism uses a Top-k Softmax routing strategy; a load balancing loss term is introduced to encourage the balance of expert utilization; and various cross-modal training samples are introduced for joint fine-tuning, only the expert module is updated in the fine-tuning, and the GOAT technology is used to reduce the parameter overhead.

Images