A query-guided adaptive three-dimensional large language model system

Abstract

The application belongs to the field of three-dimensional scene understanding and multi-modal artificial intelligence, and discloses a query-guided adaptive three-dimensional large language model system. The application proposes a query-guided adaptive cropping module and a multi-modal object-level feature enhancement module, which respectively solve the problems of redundant information processing and insufficient semantic information in three-dimensional scenes, including four steps: three-dimensional visual language alignment, query-guided adaptive cropping, multi-modal feature enhancement, and language model training optimization. Through the joint processing of multi-modal data, the application can achieve efficient and accurate three-dimensional scene understanding under the condition of no or few labeled data sets. Experimental results show that the performance of the application method on multiple standard data sets is better than that of the prior art, especially in three-dimensional question answering and scene description tasks, which exhibits significant precision improvement and robustness, especially in handling complex three-dimensional structures and fine-grained problems with higher accuracy and interpretability.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of three-dimensional scene understanding and multi-modal artificial intelligence, and particularly relates to a three-dimensional large language model system fusing a query-guided adaptive cropping and multi-modal semantic enhancement mechanism, which can be widely applied to three-dimensional scene question answering, description generation and multi-modal reasoning tasks. BACKGROUND

[0002] With the development of artificial intelligence technology, three-dimensional scene understanding has become a core research direction in the fields of autonomous driving, augmented reality and virtual reality, robot navigation, and intelligent manufacturing. Efficient and accurate analysis of spatial structures and object relationships in three-dimensional environments is of great significance for enhancing the perception, scene understanding, and environmental interaction capabilities of AI systems.

[0003] In recent years, the rapid development of large language models (LLMs) has promoted their application in multi-modal tasks beyond natural language processing, especially in three-dimensional semantic understanding tasks, showing strong reasoning and generalization capabilities. Many studies have integrated LLMs with three-dimensional point cloud data to construct 3D large language models (3D-LLMs) for scene description, question and answer reasoning, and visual positioning tasks, significantly improving the cognitive level of AI systems in complex three-dimensional environments. Although existing methods have made some progress, the current mainstream 3D-LLM architecture still faces two key problems:

[0004] Redundant object information interferes with reasoning: existing methods generally use PointNet, PointNet++, and other three-dimensional encoders to encode the entire scene into a large number of object-level tokens, only a part of which is related to the current task, and the rest of the irrelevant information introduces noise, increases the hallucination risk of the model, and affects the reasoning accuracy.

[0005] Lack of semantic-rich two-dimensional visual vector representation: three-dimensional point cloud data has advantages in spatial structure, but lacks semantic information such as color, texture, material properties, and high-level contextual relationships, limiting the model's comprehensive understanding of complex scenes. Most existing methods fail to effectively integrate multi-modal semantic vector representations from two-dimensional images, resulting in limited semantic representation capabilities.

[0006] Furthermore, current state-of-the-art (SOTA) 3D-LLM methods such as LEO and Point-LLM often simply concatenate object-level vector representations with text tokens and input them directly into a large language model, neglecting the systematic modeling of the two issues mentioned above. This hinders the further development of models in scene semantic understanding, spatial reasoning, and task execution. Therefore, there is an urgent need for a novel 3D language model architecture that integrates task-related guidance and multimodal vector representation enhancement to improve the model's ability to select key objects, enhance the semantic expression of 3D tokens, and ultimately achieve accurate perception, semantic understanding, and intelligent response to complex 3D environments. Summary of the Invention

[0007] This invention aims to address two key problems existing in current 3D Large Language Models (3D-LLMs) for 3D scene understanding tasks: First, existing models often input a large number of task-irrelevant object-level vector representations, leading to noise and illusions during inference and reducing understanding accuracy; second, relying solely on point cloud data while ignoring the rich semantic information in 2D images makes it difficult to achieve deep multimodal semantic fusion and scene understanding. Therefore, this invention proposes a novel 3D language model system that integrates a task-relevance-guided object selection mechanism with a multimodal vector representation enhancement strategy, significantly improving the accuracy and robustness of 3D scene question answering, generation, and inference tasks.

[0008] The technical solution of this invention:

[0009] A query-guided adaptive three-dimensional large language model system includes the following steps:

[0010] Step 1: 3D vision-language alignment construction;

[0011] The SentencePiece word segmenter is used to encode the system prompt text and user questions. Simultaneously, a pre-trained PointNet++ point cloud encoder is used to extract point-level vector representations of each object in the input 3D scene, obtaining the geometric vector representation of each object. Subsequently, the geometric vector representations of each object are converted into object-level vector representations at a uniform scale to obtain the 3D feature matrix. Simultaneously, predict the category label for each object in the 3D scene. ,in, Refers to the first in a three-dimensional scene One object, The number of objects in the 3D scene. The vector dimension is matched to the input dimension of the large language model; after the three-dimensional feature matrix undergoes query-guided adaptive pruning in step 2 and multimodal object-level vector representation enhancement in step 3, it is fed into the large language model along with the system prompt text and the user question code to obtain the language response sequence. ;

[0012] Step 2: Query-guided adaptive pruning;

[0013] (2.1) Question representation generation: encode the user question using a frozen BERT encoder to get the query semantic vector Meanwhile, convert the class label of each object into a class semantic vector ;

[0014] (2.2) Semantic relevance calculation: calculate the cosine similarity between the class semantic vector and the query semantic vector to form a semantic relevance list:

[0015]

[0016] (2.3) Global task-guided modeling: construct a learnable global task query vector , through its cross-attention with the query semantic vector , two layers of self-attention and pooling operation, get the pruning ratio factor , calculate the number of objects that need to be retained :

[0017]

[0018]

[0019] where, is the total number of objects in the current three-dimensional scene; represents a linear layer for linear transformation of the input vector; represents a pooling operation for aggregating vector representations; represents two self-attention layers for capturing the correlation between elements within the sequence; represents a cross-attention layer; represents the rounding operation;

[0020] (2.4) Based on the similarity ranking results, find the top object-level vector representations most relevant to the task in the three-dimensional feature matrix;

[0021] Step 3: Multi-modal object-level vector representation enhancement;

[0022] (3.1) Point-level two-dimensional vector representation extraction: first extract the pixel-level vector representation from each multi-view image of the three-dimensional scene using a pre-trained 2D encoder and respectively the height and width of the multi-view image, is the number of channels of the pixel-level vector representation; the point cloud data corresponding to the three-dimensional scene is projected to the image pixel points through the camera intrinsic and extrinsic parameters to form the pixel-level vector representation and the corresponding pixel vector representation is averaged under multiple views to form the point-level two-dimensional vector representation , is the number of points in the point cloud data corresponding to the three-dimensional scene:

[0023]

[0024] wherein, is the number of multi-view images under the three-dimensional scene;

[0025] (3.2) Object-level two-dimensional vector representation fusion and alignment: using the object mask corresponding to the 3D scene, the point-level two-dimensional vector representation is aggregated into the object-level two-dimensional vector representation and mapped to the dimension matching the large language model through the linear mapping matrix :

[0026]

[0027] (3.3) Cross-modal fusion operation: taking the three-dimensional feature matrix as the query vector, the mapped object-level two-dimensional vector representation as the key, and fusing into the fusion vector representation through the cross-attention mechanism:

[0028]

[0029] wherein, is the mapping matrix in cross-attention, represent the query vector, key vector and value vector in cross-attention calculation, denotes the normalization function; then through the linear layer, the normalization random inactivation and the residual connection, and then the normalization operation is performed again, finally according to the value calculated in step (2.3) , select object-level vector representations most relevant to the task from the three-dimensional feature matrix to obtain the enhanced three-dimensional feature matrix

[0030] Step 4: Language modeling and training optimization;

[0031] (4.1) Input construction: concatenating the system prompt text and user question encoded by the SentencePiece tokenizer and the three-dimensional feature matrix to form a unified multi-modal input sequence, and then inputting into the pre-trained large language model to generate a language response ;

[0032] (4.2) Training objective function definition: for the language response sequence , the conditional language modeling loss function is defined as:

[0033]

[0034] wherein, denotes a batch size, denotes the length of the target response sequence, denotes the current predicted time step, denotes the position of the current processed data in the batch, includes the system prompt text, the user question and the enhanced three-dimensional feature matrix, denotes the language response result of the current batch and the current predicted time step, denotes the language response result before the current time step in the current batch, denotes the conditional probability distribution parameterized by the parameter .

[0035] Advantages of the present application:

[0036] (1) The query-guided adaptive pruning module aims to solve the problem of excessive objects and insufficient task relevance in existing three-dimensional large language models. According to the user's task instructions or question content, QGAP automatically evaluates the relevance between each object-level vector representation in the scene and the user's question. The system interacts with the user's question by introducing a learnable global task query vector, and adaptively determines the number of object-level vector representations that need to be retained. Finally, the model will only retain the object-level vector representations closely related to the task, and eliminate redundant or irrelevant information, thereby reducing reasoning noise and hallucinations, and improving the understanding and reasoning accuracy of the model in complex scenes.

[0037] (2) The multi-modal object-level vector enhancement enhancement module aims to solve the problem that existing methods only rely on point cloud data and lack rich semantic information. MOFE projects the semantic vector representation in the multi-view image into the three-dimensional space by fusing two-dimensional images and three-dimensional point cloud information. On this basis, the two-dimensional vector representation is aggregated to the object level, and then fused with the object-level vector representation filtered by the QGAP module through cross-attention calculation, so that the finally generated enhanced three-dimensional feature matrix has both geometric details and semantic information. Through this multi-modal vector representation enhancement mechanism, the model can better understand the material, color, semantic attributes and contextual relationships of objects, and achieve more rich and accurate three-dimensional scene understanding.

[0038] (3) The adaptive three-dimensional large language model system based on query guidance (3D-SceneQ), the overall system 3D-SceneQ of the application is based on the above two key modules, and a novel adaptive three-dimensional large language model system based on query guidance is constructed. First, the QGAP module adaptively filters the object-level vector representation according to the task requirements, so as to ensure that the input vector representation data is more compact and task-related. Then, the MOFE further fuses two-dimensional semantic information and three-dimensional geometric vector representation to generate a three-dimensional feature matrix with rich semantics. Finally, the enhanced three-dimensional feature matrix, system prompt text and user question are input into the large language model together to complete the three-dimensional scene question answering, description and reasoning task. Through this design, 3D-SceneQ not only effectively reduces the noise and illusion problem, but also enhances the semantic understanding and reasoning ability of the model for complex scenes, and shows excellent accuracy and robustness in three-dimensional question answering, three-dimensional description generation and interactive reasoning tasks. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a 3D-SceneQ framework overview diagram;

[0040] Figure 2 is a query-guided adaptive pruning module (QGAP) flowchart;

[0041] Figure 3 is a multi-modal object-level vector representation enhancement module (MOFE) flowchart. DETAILED DESCRIPTION

[0042] The specific embodiments of the application will be further described below in combination with the drawings and technical solutions.

[0043] EMBODIMENT

[0044] An adaptive three-dimensional large language model system 3D-SceneQ based on query guidance, which combines an adaptive pruning mechanism guided by queries and a multi-modal semantic enhancement strategy to improve semantic understanding and language reasoning ability in complex three-dimensional scenes, specifically includes the following steps:

[0045] Step 1: Three-dimensional visual-language alignment construction;

[0046] As shown in Figure 1 , the SentencePiece tokenizer is used to encode the system prompt text and user question, and the pre-trained point cloud encoder PointNet++ is used to extract the point-level vector representation of each object in the input three-dimensional scene to obtain the geometric vector representation of each object; then the geometric vector representation of each object is converted into a unified scale object-level vector representation to obtain a three-dimensional feature matrix , and the class label of each object in the three-dimensional scene is predicted wherein, refers to the i-th object in the three-dimensional scene, is the number of objects in the three-dimensional scene, is the vector dimension matching the input dimension of the large language model; after the three-dimensional feature matrix is subjected to query-guided adaptive pruning in step 2 and multi-modal object-level vector representation enhancement in step 3, it is jointly sent into the large language model together with the system prompt text and user question encoding to obtain a language response sequence ;

[0047] Step 2: Query-guided adaptive pruning.

[0048] (2.1) As shown in FIG. 1, question representation generation: encode the user question using a frozen BERT encoder to obtain a query semantic vector Figure 2 , while converting the class label of each object into a class semantic vector through the BERT encoder .

[0049] (2.2) Semantic correlation calculation: calculate the cosine similarity between the class semantic vector and the query semantic vector to form a semantic correlation list:

[0050]

[0051] (2.3) Global task guidance modeling: construct a learnable global task query vector , through cross-attention with the query semantic vector , two layers of self-attention and pooling operation, to obtain a pruning ratio factor , and calculate the number of objects to be retained :

[0052]

[0053]

[0054] wherein, is the total number of objects in the current three-dimensional scene; represents a linear layer for linear transformation of the input vector; represents a pooling operation for aggregating vector representation; represents two self-attention layers for capturing the correlation between elements in the sequence; represents a cross-attention layer; represents an integer operation;

[0055] ​(2.4) Based on the similarity ranking result, find the top object-level vector representation in the three-dimensional feature matrix that is most relevant to the task;

[0056] Step 3: Multimodal object-level vector representation enhancement;

[0057] (3.1) Point-level two-dimensional vector representation extraction: as shown in Figure 3 , first extract the pixel-level vector representation of each multi-view image from the three-dimensional scene corresponding multi-view image using the pre-trained 2D encoder , where and are the height and width of the multi-view image, is the number of channels of the pixel-level vector representation; by camera internal and external parameters, project the point cloud data corresponding to the three-dimensional scene to the image pixel and average the corresponding pixel vector representation under multiple views to form the point-level two-dimensional vector representation , is the number of points in the point cloud data corresponding to the three-dimensional scene:

[0058]

[0059] where is the number of multi-view images under the three-dimensional scene.

[0060] (3.2) Object-level two-dimensional vector representation fusion and alignment: using the object mask corresponding to the 3D scene, the point-level two-dimensional vector representation is aggregated into the object-level two-dimensional vector representation , and mapped to the dimension matching the large language model through the linear mapping matrix :

[0061]

[0062] (3.3) Cross-modal fusion operation: taking the three-dimensional feature matrix as the query vector, the mapped object-level two-dimensional vector representation as the key, and through the cross-attention mechanism, it is fused into the fusion vector representation:

[0063]

[0064] where is the mapping matrix in cross-attention, represent the query vector, key vector and value vector in cross-attention calculation, denotes the normalization function; then through the linear layer, the normalization random inactivation and the residual connection, then again the normalization operation, finally according to the values, picking object-level vector representations most relevant to the task in the three-dimensional feature matrix, obtaining an enhanced three-dimensional feature matrix ;

[0065] Step 4: Language modeling and training optimization

[0066] (4.1) Input construction: The system prompt text and user question coded by the SentencePiece tokenizer and the three-dimensional feature matrix are spliced to form a unified multi-modal input sequence, which is then input into a pre-trained large language model to generate a language response ;

[0067] (4.2) Training objective function definition: For the language response sequence , the conditional language modeling loss function is defined as:

[0068]

[0069] wherein, represents a batch size, represents the length of the target response sequence, represents the current predicted time step, represents the position of the current processed data in the batch, includes the system prompt text, the user question, and the final enhanced three-dimensional feature matrix, represents the language response result of the current batch and the current predicted time step, represents the language response result before the current time step in the current batch, represents the conditional probability distribution parameterized by the parameter .

[0070] Experimental setup and effect verification:

[0071] Experimental data set: In order to evaluate the 3D-SceneQ model proposed in the present application, multiple three-dimensional visual-language data sets are used, covering different task alignment and instruction fine-tuning processes, as follows:

[0072] (1) 3D vision-language alignment stage: The model is trained to bridge the gap between three-dimensional scene representations and natural language, using the following three types of descriptive data: Object-level descriptions: Dataset from Cap3D, providing fine-grained textual descriptions from Objaverse, describing the details of different three-dimensional objects. Object in context refers to representation: Including ScanScribe and ReferIt3D datasets, which provide information about object references in three-dimensional scenes through user questions, and use LLM-generated annotations to enhance context diversity. Scene-level summaries: Dataset from 3RScan, capturing global information such as object layout, spatial relationships, and scene semantics.

[0073] (2) Instruction fine-tuning stage: In this stage, the model is fine-tuned with instructions to improve its ability to understand, reason, and respond to natural language instructions in three-dimensional environments. Datasets based on ScanNet and 3RScan are used, covering open three-dimensional scene description and question-answering tasks. The model needs to generate scene descriptions or answer questions about object attributes, locations, and relationships.

[0074] (3) Evaluation metrics: To evaluate the model's performance in different tasks, a series of standard evaluation metrics are used, as follows:

[0075] 1) Language generation task: For language generation tasks, metrics such as CIDEr, BLEU, METEOR, and ROUGE are reported, measuring the relevance, fluency, and overlap of generated text with reference text.

[0076] 2) Open-ended generation task: In open-ended generation tasks, where multiple correct answers may exist, a sentence similarity metric is also used to better capture semantic alignment. Question-answering task: For question-answering tasks that require factual accuracy, EM@1 (Top-1 accuracy) is reported, reflecting the proportion of model predictions that perfectly match the true answers.

[0077] Implementation details: In terms of implementation, Vicuna-7B is used as the backbone network for the language model, directly using the pre-trained version of the LEO model as initialization. The components of the model are as follows: 3D encoder, two-dimensional semantic segmentation model used in MOFE, language tokenizer, and language backbone network, as well as BERT encoder in QGAP, all of which remain frozen during training. LoRA method is used to efficiently fine-tune Vicuna-7B, with a low-rank setting of r=16. Other trainable parameters, including QGAP, MOFE, and object-to-text mapping layers, are trained from scratch. The AdamW optimizer is used in the training process, with a learning rate of , for 10 rounds of training. All experiments were conducted on two NVIDIA A8000 GPUs.

[0078] Table 1. Performance comparison of 3D-SceneQ and current mainstream models on Scan2Cap and SQA3D benchmark tests

[0079]

[0080] where “EM@1” denotes Top-1 exact match accuracy; n-gram evaluation metrics in Scan2Cap use an IoU threshold of ≥ 0.5.

[0081] Table 2. Performance comparison of 3D-SceneQ and current mainstream models on ScanQA benchmark tests

[0082]

[0083] “EM@1” denotes Top-1 exact match accuracy.

[0084] Table 3. Performance comparison of 3D-SceneQ and current mainstream models on Scan2Cap and SQA3D benchmark tests

[0085]

[0086] “EM@1” denotes Top-1 exact match accuracy; n-gram evaluation metrics in Scan2Cap use an IoU threshold of ≥ 0.5.

[0087] Table 4. Performance comparison of 3D-SceneQ and current mainstream models on ScanQA benchmark tests

[0088]

[0089] “EM@1” denotes Top-1 exact match accuracy.

[0090] As shown in Tables 1 and 2, 3D-SceneQ achieves SOTA performance in Scan2Cap, ScanQA, and SQA3D tasks, surpassing strong task-specific benchmark models and fine-tuned specialized models. For example, in the Scan2Cap task, 3D-SceneQ improves METEOR to 30.5 and ROUGE to 65.3, surpassing the previous best model by 2.6 and 7.2 points; in the ScanQA task, CIDEr is improved to 104.7 and BLEU-4 to 13.6, reflecting more relevant and fluent answers generated by the model; on the SQA3D test set, EM@1 reaches 67.0%, 17 percentage points higher than the strongest pre-posed method, demonstrating the ability to generate accurate and contextually rich outputs.

[0091] Key Module Effectiveness Verification: By comparing with several state-of-the-art (SOTA) methods, the superior performance of the proposed method in 3D scene understanding tasks was verified. Specifically, a series of ablation experiments were conducted to deeply analyze the individual and combined effects of the two core modules, QGAP and MOFE. Tables 3 and 4 show the performance comparison of QGAP and MOFE. Using the removal of components related to the agent representation in the LEO architecture as a baseline model for comparison, the following four main conclusions were drawn:

[0092] Random pruning helps reduce redundancy but may lose important information. The RandFilter variant randomly discards 40% of objects, but this indiscriminate compression causes Scan2Cap's CIDEr to drop from 72.4 to 52.3. This result suggests that reducing redundant information helps inference effectiveness, but pruning must be based on task relevance.

[0093] QGAP improves description quality and inference accuracy. By learning the filtering ratio based on the user's question, QGAP retains objects that are highly relevant to the task, improving Scan2Cap CIDEr to 78.4 and SQA3D EM@1 to 70.1, outperforming the baseline model and random pruning.

[0094] MOFE improves overall performance, especially on the ScanQA task. By fusing geometric, visual, and speech signals, MOFE improves Scan2Cap CIDEr to 77.8, showing a significant performance boost, particularly in the ScanQA task.

[0095] The synergy between QGAP and MOFE delivers the most balanced performance improvements. 3D-SceneQ combines QGAP’s task-related clipping with MOFE’s enhancements, achieving significant performance improvements across all benchmarks, such as a 5.7-point improvement in Scan2Cap CIDEr and a 20.0-point improvement in SQA3D EM@1.

Claims

1. A query-guided adaptive three-dimensional large language model system, characterized in that, Comprising the following steps: Step 1: Three-dimensional visual-linguistic alignment construction; The system prompt text and user questions are encoded using the SentencePiece tokenizer, and a pre-trained point cloud encoder PointNet++ is used to extract point-level vector representations of each object in the input three-dimensional scene, obtaining the geometric vector representation of each object; Subsequently, the geometric vector representation of each object is converted into a unified scale object-level vector representation to obtain a three-dimensional feature matrix At the same time, the class label of each object in the three-dimensional scene is predicted Wherein, Refers to the th object in the three-dimensional scene, The number of objects in the three-dimensional scene, The vector dimension matching the input dimension of the large language model; Step 2: Query-guided adaptive pruning; (2.1) Question representation generation: encode the user question using the frozen BERT encoder to obtain a query semantic vector At the same time, the category label of each object is converted into a category semantic vector through the BERT encoder ; (2.2) Semantic relevance calculation: calculate cosine similarity of category semantic vector and query semantic vector, form semantic relevance list: ​ ； (2.3) Global task guidance modeling: build a learnable global task query vector , through its cross attention with the query semantic vector , two layers of self-attention and pooling operation, get the pruning ratio factor , calculate the number of objects that need to be retained : ； ； wherein, is the total number of objects within the current three-dimensional scene; denotes a linear layer for linearly transforming the input vector; denotes a pooling operation for aggregating the vector representations; denotes a self-attention layer for capturing the correlation between elements within the sequence; denotes a cross-attention layer; denotes a rounding operation; (2.4) Based on the similarity ranking results, find the top object-level vector representations in the three-dimensional feature matrix that are most relevant to the task; Step 3: Multi-modal object-level vector representation enhancement; (3.1) Point-level 2D vector representation extraction: first, the pixel-level vector representation of each multi-view image is extracted from the multi-view image corresponding to the three-dimensional scene using a pre-trained 2D encoder wherein, and are the height and width of the multi-view image, respectively, is the number of channels of the pixel-level vector representation; the point cloud data corresponding to the three-dimensional scene is projected to the image pixel points by the camera intrinsic and extrinsic parameters and the corresponding pixel vector representations are averaged under multiple perspectives to form the point-level 2D vector representation , is the number of points in the point cloud data corresponding to the three-dimensional scene: ； wherein, is the number of multi-view images in a three-dimensional scene; (3.2) Object-level 2D vector representation fusion and alignment: using the object mask corresponding to the 3D scene, the point-level 2D vector representation is aggregated into the object-level 2D vector representation , and mapped to the dimension matching the large language model through a linear mapping matrix : ； (3.3) Cross-modal fusion operation: with the three-dimensional feature matrix as the query vector, the mapped object-level two-dimensional vector representation as the key, fused into a fused vector representation by cross-attention mechanism: ； wherein, is a mapping matrix in cross attention, respectively represent query vector, key vector and value vector in cross attention calculation, denotes a normalization function; then through a linear layer, the normalization random inactivation is connected with the residual connection, and then the normalization operation is performed again, and finally the value calculated in step (2.3) is selected in the three-dimensional feature matrix object-level vector representation most relevant to the task, to obtain an enhanced three-dimensional feature matrix ; Step 4: Language modeling and training optimization.

2. The query-guided adaptive three-dimensional large language model system according to claim 1, wherein, The specific implementation process of Step 4 is as follows: (4.1) Input construction: system prompt text and user question encoded by the SentencePiece tokenizer and the three-dimensional feature matrix are concatenated to form a unified multi-modal input sequence, which is then fed into a pre-trained large language model to generate a language response ; (4.2) Training objective function definition: for the language response sequence The conditional language modeling loss function is defined as: ； wherein, denotes a batch size, denotes the length of the target response sequence, denotes the current predicted time step, denotes the position of the currently processed data in the batch, comprises system prompt text, user questions and the final enhanced three-dimensional feature matrix, denotes the language response result for the current batch, the current predicted time step, denotes the language response result for the current batch, before the current time step, denotes the conditional probability distribution parameterized by the parameters parameters.

Images