Multimodal training data synthesis system based on semantic space world model

Abstract

This invention discloses a multimodal training data synthesis system based on a semantic space world model, belonging to the field of artificial intelligence technology. The invention includes a discrete visual vector quantizer, a unified sequence modeling network, and a visual representation recovery module. It skips pixel-level generation and multi-step denoising, reducing the average generation time per sample and achieving millisecond-level controllable synthesis. Based on a large language model, it unifies the modeling of text and visual sequences, accurately understanding complex long text instructions. The results are significantly faster than manual data collection and closed-source API calls. The output visual representations can be directly used for multimodal large-scale model training, covering scenarios such as instruction fine-tuning, reinforcement learning, streaming perception, and dangerous scene synthesis.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a multimodal training data synthesis system based on a semantic space world model. Background Technology

[0002] With the widespread application of multimodal large-scale models in fields such as visual-text understanding, video analysis, embodied intelligence, streaming perception, and intelligent early warning, the demand for large-scale, highly consistent, strongly instruction-aligned, and scene-controllable multimodal training data continues to grow. For some special scenarios, dangerous scenarios, long-term continuous scenarios, and world state transition data required for reinforcement learning, it is impossible to obtain them through manual collection or actual filming. Therefore, it is necessary to analyze multimodal training data synthesis systems based on semantic space world models.

[0003] Existing technologies, such as the invention application patent with announcement number CN121503641A, disclose an integrated training and inference system and method based on multimodal data fusion. The system includes a multimodal training data analysis module, a multimodal data model training module, and a multimodal data training model selection module. It intelligently matches a general training model for multimodal training data based on information about the types and application scenarios of the multimodal training data, combined with artificial intelligence algorithms and the feature information of general multimodal training models. This enables intelligent selection of large multimodal data models based on the types of multimodal training data and the application scenarios of large models. Furthermore, it performs parameterized evaluation of the training results of the multimodal data-trained model by executing the multimodal data training program, using multimodal validation data, and combining large model evaluation tools. This achieves refined evaluation of the training quality of the multimodal data fusion large model based on the training feature index parameters of the large model.

[0004] Existing technologies can meet the basic requirements for multimodal training data synthesis systems based on semantic space world models, but they also have some potential defects and challenges, specifically in the following aspects: First, the model has a large number of parameters, many inference steps based on diffusion architecture, slow data synthesis speed, and high computing power and economic costs. Second, the text conditional encoder has a small capacity and weak comprehension ability, making it difficult to accurately follow complex and long text instructions, resulting in poor consistency between the generated content and the text instructions. Third, mainstream high-performance models are closed-source and rely on expensive API calls, while open-source solutions are limited to niche areas such as vision-language-action VLA, and lack versatility. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal training data synthesis system based on a semantic space world model, which solves the problems existing in the background technology.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a multimodal training data synthesis system based on a semantic space world model, including a discrete visual vector quantizer, a unified sequence modeling network, and a visual representation recovery module.

[0007] The discrete visual vector quantizer is used to extract continuous visual features from the input image through a pre-trained visual encoder and map them into a discrete visual token sequence through a vector quantization codebook.

[0008] The unified sequence modeling network is used to perform autoregressive joint modeling of text token sequences and discrete visual token sequences, and predict subsequent visual token sequences based on text instructions or reference image conditions.

[0009] The visual representation recovery module is used to restore the predicted discrete visual token sequence into a visual representation sequence that can be directly used for training multimodal large models.

[0010] The system achieves low-cost, high-speed, and strong instruction-following capability synthesis of multimodal training data by directly synthesizing visual representation sequences in the semantic space, skipping pixel-level image / video generation.

[0011] Furthermore, the discrete visual vector quantizer includes: a pre-trained visual encoder SigLip2 for extracting two-dimensional continuous visual feature maps; a vector quantization codebook (VQ codebook) for mapping continuous visual features to discrete tokenID sequences; and a two-layer perceptron for projecting continuous visual features onto the quantization space and then performing nearest-neighbor quantization.

[0012] Furthermore, the specific analysis method of the unified sequence modeling network is as follows: a pre-trained large language model is used as the backbone, text token embedding and discrete visual token embedding are uniformly mapped to the same latent space dimension and then spliced ​​together for modeling, and an autoregressive causal mask is used to complete the visual token sequence prediction.

[0013] Furthermore, the visual representation recovery module uses the following specific analysis method: it employs a multi-layer residual network ResNet structure to recover the codeword embedding sequence corresponding to the discrete visual token into a two-dimensional feature map, and outputs a visual representation sequence with the same dimension as the original visual encoder.

[0014] Furthermore, the training process of the system is divided into three stages: the first stage: jointly training the discrete visual vector quantizer and the visual representation recovery module, with the MSE loss of continuous visual features and the recovered visual representation as the optimization objective, and freezing all parameters after training.

[0015] The second stage involves training a unified sequence modeling network using supervised fine-tuning, with the cross-entropy loss from visual token sequence prediction as the optimization objective.

[0016] The third stage involves using PPO-based reinforcement learning to optimize the unified sequence modeling network, rewarding it with the accuracy of multimodal large model tasks, and introducing KL divergence constraints to limit policy bias.

[0017] Furthermore, the specific analysis process for the jointly trained discrete visual vector quantizer and visual representation recovery module is as follows: 001. Input Image - Visual Encoder - Continuous Visual Features.

[0018] 002, Continuous visual features - vector quantization codebook - discrete visual token.

[0019] 003. Discrete visual token - Visual representation recovery module - The recovered visual representation sequence is used to calculate the loss by combining the recovered visual representation sequence with continuous visual features.

[0020] Furthermore, the specific analysis process for training the unified sequence modeling network in a supervised fine-tuning manner is as follows: A. Discrete visual token sequence + text sequence - unified sequence modeling network.

[0021] B. Autoregressive prediction of target visual token sequence.

[0022] Furthermore, the specific analysis process of the unified sequence modeling network optimized by reinforcement learning based on PPO is as follows: Step 1: Generate discrete visual token sequences using the supervised fine-tuned network.

[0023] Step 2: Obtain the generated sequence through the visual representation recovery module.

[0024] Step 3: Calculate the reward: whether the multimodal large model can correctly understand the generated sequence.

[0025] Furthermore, the specific optimization objective of the unified sequence modeling network using PPO-based reinforcement learning is to maximize the expected reward and policy bias of the generated visual token sequence on the task to ensure training stability.

[0026] Furthermore, the expected reward and constraint policy offset, in the reinforcement learning fine-tuning stage, use the multimodal model's answer accuracy on the evaluation set as the reward signal, use the supervised fine-tuning frozen model as the reference policy, and maintain generation stability through KL divergence constraints.

[0027] The beneficial effects of this invention are as follows: First, fast synthesis speed: skipping pixel-level generation and multi-step denoising, reducing the average generation time of a single sample, and achieving millisecond-level controllable synthesis; 2. Strong command compliance capability: Based on a unified model of large language model, it can accurately understand complex and long text commands. III. Significantly reduced costs: The cost of synthesizing a single sample is reduced, far lower than manual collection and closed-source API calls; IV. High versatility: The output visual representation can be directly used for training of multimodal large models, covering scenarios such as instruction fine-tuning, reinforcement learning, streaming perception, and dangerous scene synthesis. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the system structure connection of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Reference Figure 1 As shown, the present invention provides a multimodal training data synthesis system based on a semantic space world model, including a discrete visual vector quantizer, a unified sequence modeling network, and a visual representation recovery module.

[0032] The discrete visual vector quantizer is used to extract continuous visual features from the input image through a pre-trained visual encoder and map them into a discrete visual token sequence through a vector quantization codebook.

[0033] It should be noted that the discrete visual vector quantizer has the following internal architecture: 1. A pre-trained visual encoder (SigLip2) extracts two-dimensional continuous visual feature maps; 2. A vector quantization codebook (VQ codebook) maps continuous features to discrete tokenID sequences.

[0034] Input: Real image or reference image; Output: Discrete visual token sequence.

[0035] Detailed technical parameters: The pre-trained visual encoder uses SigLip2 as the image feature extraction backbone network. The input image resolution is preferably 384×384, and the patch size is preferably 16×16, corresponding to 24×24 visual token positions. The dimension of the continuous visual features output by the visual encoder is preferably 1152. The codebook capacity of the vector quantization codebook is preferably set to 8192 codewords, and the embedding dimension of each codeword is preferably 32. To reduce the difficulty of discrete modeling, the continuous visual features are first projected onto the 32-dimensional quantization space through a two-layer perceptron, and then nearest neighbor quantization is performed. The hidden layer dimension of the two-layer perceptron is preferably 512, and the activation function is preferably GELU.

[0036] The unified sequence modeling network is used to perform autoregressive joint modeling of text token sequences and discrete visual token sequences, and predict subsequent visual token sequences based on text instructions or reference image conditions.

[0037] It should be noted that the internal architecture of the unified sequence modeling network is a pre-trained large language model.

[0038] Input: Text token sequence + discrete visual token sequence.

[0039] Output: Predict the subsequent visual token sequence.

[0040] Detailed technical parameters: In one implementation, the unified sequence modeling network uses a pre-trained large language model as its backbone. The preferred number of network layers is 32, the preferred hidden layer dimension is 4096, and the preferred number of attention heads is 32. Text token embeddings and discrete visual token embeddings are uniformly mapped to the same latent space dimension before being concatenated for modeling. The maximum sequence length is preferably set to 4096, which may include text instruction tokens, scene script tokens, reference visual tokens, and target visual tokens. During training, an autoregressive causal mask is used, enabling the model to predict subsequent visual tokens based solely on historical tokens.

[0041] The visual representation recovery module is used to restore the predicted discrete visual token sequence into a visual representation sequence that can be directly used for training multimodal large models.

[0042] It should be noted that the visual representation recovery module has an internal architecture of a multi-layer ResNet.

[0043] Input: The predicted discrete visual token sequence.

[0044] Output: Visual representation sequence.

[0045] Detailed technical parameters: The visual representation recovery module adopts a ResNet structure composed of 8 stacked residual blocks. The input is the codeword embedding sequence corresponding to the discrete visual token. It is first recovered into a two-dimensional feature map, and then output as the target visual representation sequence after convolution mapping. The convolution kernel size in each residual block is preferably 3×3, and the number of channels is preferably 512. The visual representation dimension output by the recovery module is preferably consistent with the output dimension of the original visual encoder, which is 1152 dimensions, so as to be directly used as the training input for subsequent multimodal large models.

[0046] The system achieves low-cost, high-speed, and strong instruction-following capability synthesis of multimodal training data by directly synthesizing visual representation sequences in the semantic space, skipping pixel-level image / video generation.

[0047] In the above embodiments, the discrete visual vector quantizer includes: a pre-trained visual encoder SigLip2 for extracting two-dimensional continuous visual feature maps; a vector quantization codebook (VQ codebook) for mapping continuous visual features to discrete token ID sequences; and a two-layer perceptron for projecting continuous visual features onto the quantization space and then performing nearest neighbor quantization.

[0048] In the above embodiments, the specific analysis method of the unified sequence modeling network is as follows: a pre-trained large language model is used as the backbone, text token embedding and discrete visual token embedding are uniformly mapped to the same latent space dimension and then spliced ​​together for modeling, and an autoregressive causal mask is used to complete the visual token sequence prediction.

[0049] In the above embodiments, the visual representation recovery module specifically analyzes the following method: using a multi-layer residual network ResNet structure, the codeword embedding sequence corresponding to the discrete visual token is recovered into a two-dimensional feature map, and a visual representation sequence with the same dimension as the original visual encoder is output.

[0050] In the above embodiment, the training process of the system is divided into three stages: the first stage: jointly training the discrete visual vector quantizer and the visual representation recovery module, with the MSE loss of continuous visual features and the recovered visual representation as the optimization target, and freezing all parameters after training.

[0051] The second stage involves training a unified sequence modeling network using supervised fine-tuning, with the cross-entropy loss from visual token sequence prediction as the optimization objective.

[0052] The third stage involves using PPO-based reinforcement learning to optimize the unified sequence modeling network, rewarding it with the accuracy of multimodal large model tasks, and introducing KL divergence constraints to limit policy bias.

[0053] In the above embodiments, the specific analysis process of the jointly trained discrete visual vector quantizer and visual representation recovery module is as follows: 001, Input image - visual encoder - continuous visual features.

[0054] 002, Continuous visual features - vector quantization codebook - discrete visual token.

[0055] 003. Discrete visual token - Visual representation recovery module - The recovered visual representation sequence is used to calculate the loss by combining the recovered visual representation sequence with continuous visual features.

[0056] It should be noted that the training hyperparameter settings are as follows: In one implementation, Phase 1 training is used to jointly optimize the discrete visual vector quantizer and the visual representation recovery module. The training batch size is preferably 256, the optimizer is preferably AdamW, the initial learning rate is preferably 1×10^-4, and the weight decay is preferably 0.01. The number of training epochs is preferably 20, the learning rate scheduling strategy is preferably cosine decay, and a warmup of 5% of the training steps is set.

[0057] Loss function: MSE loss.

[0058] Output: Learned discrete visual vector quantizer suitable for discrete modeling, and visual representation recovery module suitable for recovery.

[0059] Note: After training is complete, all parameters of these two modules are frozen for use in subsequent stages.

[0060] In the above embodiments, the specific analysis process of training the unified sequence modeling network in a supervised fine-tuning manner is as follows: A. Discrete visual token sequence + text sequence - unified sequence modeling network.

[0061] B. Autoregressive prediction of target visual token sequence.

[0062] It should be noted that the training hyperparameter settings are as follows: In one implementation, the unified sequence modeling network is trained using supervised fine-tuning in stage two. The training batch size is preferably 128, the optimizer is preferably AdamW, the initial learning rate is preferably 2×10^-5, and the weight decay is preferably 0.1. The maximum number of training steps is preferably 100,000 steps.

[0063] Loss function: Visual token sequence prediction loss.

[0064] Output: Learn to stably generate discrete visual tokens.

[0065] In the above embodiment, the specific analysis process of using PPO-based reinforcement learning to optimize the unified sequence modeling network is as follows: Step 1: Use the supervised fine-tuned network to generate discrete visual token sequences.

[0066] Step 2: Obtain the generated sequence through the visual representation recovery module.

[0067] Step 3: Calculate the reward: whether the multimodal large model can correctly understand the generated sequence.

[0068] It should be noted that the training hyperparameter settings are as follows: In one implementation, stage three employs a PPO-based reinforcement learning optimization method to further optimize the supervised fine-tuned unified sequence modeling network. The training batch size is preferably 64, the number of candidate sequences generated per sample is preferably 4, the optimizer is preferably AdamW, and the learning rate is preferably 1×10^-6.

[0069] To prevent the generated policy from deviating too much from the initial policy during supervised fine-tuning during reinforcement learning training, a reference policy constraint term is introduced, with a preferred weight coefficient of 0.02. During each reinforcement learning update, the network from the frozen copy of the network from the supervised fine-tuning phase is preferably used as the reference policy.

[0070] The specific calculation of the reward function in the reinforcement learning stage: On the MMMU benchmark, 10 multiple-choice questions are extracted in each step. A pre-trained llava1.5-7b model uses the visual sequence generated by the semantic space world model. At this time, the input of the semantic space world model is the image corresponding to the benchmark question. After passing through the text caption generated by the pre-trained llava1.5-7b model, the correct answer is used as the reward of 0-1.

[0071] Implementation of policy offset constraint: To ensure training stability during the reinforcement learning phase and avoid excessive drift of the generated policy relative to the initial policy during the supervised fine-tuning phase, in one implementation, a KL divergence constraint relative to the reference policy is introduced. The optimization objective during the reinforcement learning phase can be expressed as: L_RL=-E[R]+η D_KL(π_θ||π_ref), where R is the reward value, π_θ is the current policy to be optimized, π_ref is the reference policy obtained and frozen during the supervised fine-tuning phase, and η is the policy offset penalty coefficient, preferably 0.02.

[0072] In the above embodiments, the specific optimization objective of the unified sequence modeling network using PPO-based reinforcement learning is to maximize the expected reward and policy bias of the generated visual token sequence on the task to ensure training stability.

[0073] In the above embodiments, the expected reward and the constraint policy offset are used in the reinforcement learning fine-tuning stage. The correct answer rate of the multimodal model on the evaluation set is used as the reward signal, the supervised fine-tuning freeze model is used as the reference policy, and the generation stability is maintained by KL divergence constraint.

[0074] General experimental conditions for the embodiments: In one embodiment, the model training and testing of the present invention were completed on a server with 8 NVIDIA A100 80GB GPUs, an Intel Xeon Gold 6348 CPU, an Ubuntu 22.04 system environment, a PyTorch 2.2 deep learning framework, and CUDA version 12.1.

[0075] Training data sources include: publicly available image and text datasets, publicly available video description datasets, some self-built scene data, and text script data automatically expanded by a large language model according to the needs of the target scene. Public datasets include COCO, Visual Genome, WebVid, and Ego4D.

[0076] During the evaluation, we used accuracy as the metric from 0 to 100, and the evaluation results were judged by calling the GPT5 API.

[0077] Multimodal large model instruction fine-tuning data synthesis: Existing multimodal large model instruction fine-tuning data generation often relies on hand-captured / pre-trained images and video models based on world models. Our solution offers a more powerful alternative with lower feasibility and cost compared to hand-captured data, and better performance and lower overhead compared to traditional synthesis-based solutions. For example, our solution costs $0.002 to generate a training sample, while the cost of a hand-captured solution is $3, and the cost of calling the pre-trained model generation API is $0.5.

[0078] Multimodal large-scale model reinforcement learning training: Existing multimodal large-scale model reinforcement learning techniques are generally offline RL because they cannot obtain real-world state changes in online RL unless deployed on a real device. For offline RL, it is often necessary to manually construct real image datasets, which is costly. For online RL, our technology makes this possible with a millisecond-level generation speed. For example, our solution generates a training sample in an average of 0.7 seconds, while the generation time based on a pre-trained generative model is 17.2 seconds.

[0079] Domain Adaptation of Streaming Multimodal Large Models: To adapt streaming multimodal large models to specific scenarios, such as travel safety warnings, training requires datasets of real-time hazard assessments from hours of continuous video. Firstly, labeled crime datasets are difficult to obtain due to their destructive nature. Secondly, the long duration and precise, controllable first-person perspective of the content increase the cost of dataset acquisition. Our technology enables the synthesis of such data. Compared to data synthesized using the closed-source Veo3-pro model, our synthesized data significantly improves the performance of streaming multimodal large models on travel safety tasks. For example, on the internal travel safety benchmark, our method achieves 59.2 vs. Veo3-pro's 21.8.

[0080] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A multimodal training data synthesis system based on a semantic space world model, characterized in that, The system consists of three main modules: a discrete visual vector quantizer, a unified sequence modeling network, and a visual representation recovery module. The discrete visual vector quantizer is used to extract continuous visual features from the input image through a pre-trained visual encoder and map them into a discrete visual token sequence through a vector quantization codebook. The unified sequence modeling network is used to perform autoregressive joint modeling of text token sequences and discrete visual token sequences, and predict subsequent visual token sequences based on text instructions or reference image conditions. The visual representation recovery module is used to restore the predicted discrete visual token sequence into a visual representation sequence that can be directly used for training a multimodal large model. The system achieves low-cost, high-speed, and strong instruction-following capability synthesis of multimodal training data by directly synthesizing visual representation sequences in the semantic space, skipping pixel-level image / video generation.

2. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The discrete visual vector quantizer includes: a pre-trained visual encoder SigLip2 for extracting two-dimensional continuous visual feature maps; a vector quantization codebook (VQ codebook) for mapping continuous visual features to discrete token ID sequences; and a two-layer perceptron for projecting continuous visual features onto the quantization space and then performing nearest-neighbor quantization.

3. The multimodal training data synthesis system based on a semantic space world model according to claim 2, characterized in that, The specific analysis method of the unified sequence modeling network is as follows: Using a pre-trained large language model as the backbone, text token embeddings and discrete visual token embeddings are uniformly mapped to the same latent space dimension and then concatenated for modeling. Autoregressive causal masks are used to complete visual token sequence prediction.

4. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The visual representation restoration module uses the following specific analysis method: A multi-layer residual network ResNet structure is used to restore the codeword embedding sequence corresponding to discrete visual tokens into a two-dimensional feature map, and output a visual representation sequence with the same dimension as the original visual encoder.

5. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The training process of the system is divided into three stages: the first stage: jointly training the discrete visual vector quantizer and the visual representation recovery module, with the MSE loss of continuous visual features and the recovered visual representation as the optimization target, and freezing all parameters after training. The second stage involves training a unified sequence modeling network using supervised fine-tuning, with visual token sequence prediction cross-entropy loss as the optimization objective. The third stage involves using PPO-based reinforcement learning to optimize the unified sequence modeling network, rewarding it with the accuracy of multimodal large model tasks, and introducing KL divergence constraints to limit policy bias.

6. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The specific analysis process for the jointly trained discrete visual vector quantizer and visual representation recovery module is as follows:

001. Input image - visual encoder - continuous visual features; 002, Continuous visual features - vector quantization codebook - discrete visual token; 003. Discrete visual token - Visual representation recovery module - The recovered visual representation sequence is used to calculate the loss by combining the recovered visual representation sequence with continuous visual features.

7. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The specific analysis process for training the unified sequence modeling network using supervised fine-tuning is as follows: A. Discrete visual token sequence + text sequence - unified sequence modeling network; B. Autoregressive prediction of target visual token sequence.

8. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The specific analysis process for optimizing the unified sequence modeling network using PPO-based reinforcement learning is as follows: Step 1: Use the supervised fine-tuned network to generate a discrete visual token sequence; Step 2: Obtain the generated sequence through the visual representation restoration module; Step 3: Calculate the reward: whether the multimodal large model can correctly understand the generated sequence.

9. The multimodal training data synthesis system based on a semantic space world model according to claim 1, characterized in that, The specific optimization objective of the unified sequence modeling network using PPO-based reinforcement learning is as follows: Maximize the expected reward of the generated visual token sequence on the task and limit the policy offset to ensure training stability.

10. The multimodal training data synthesis system based on a semantic space world model according to claim 9, characterized in that, The expected reward and constraint policy offset are described. In the reinforcement learning fine-tuning stage, the correct answer rate of the multimodal model on the evaluation set is used as the reward signal, the supervised fine-tuning freeze model is used as the reference policy, and the generation stability is maintained by KL divergence constraint.

Images