Unified framework-based Chinese multi-modal data generation method
Through the multimodal data generation method under the unified framework, the information fusion, consistency and forgetting problems in multimodal data processing are solved, efficient multimodal data generation and understanding are achieved, and system performance and user experience are improved.
Patent Information
- Application Number
- CN202411225063.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-20
AI Technical Summary
Existing multimodal data processing technologies face problems such as insufficient information fusion among modals, consistency in data generation, catastrophic forgetting, scarcity and bias in training data, and cannot efficiently understand and generate multiple modal data under a unified framework.
Adopt Chinese multimodal data generation method based on a unified framework, including data preprocessing, feature extraction, feature alignment and splicing, multimodal big model generation and content integration, and extract features through pre-training models, bridge layer model alignment features, experts Weighted scores to generate content to ensure unified data format, high-quality features and modal correlation.
It improves the efficiency of information fusion between modals, consistency and quality of generated data, reduces catastrophic forgetting problems, and improves system performance and user experience.
Smart Images
Figure CN119168064A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of natural language processing and computer vision technology, and specifically to the generation of Chinese multimodal data based on a unified framework. Background Art
[0002] In today's era of rapid information development, the generation and use of text, image, video and audio data has increased dramatically, bringing huge data processing needs. Especially in the context of multimedia technology and intelligent interactive systems, how to effectively understand and generate multimodal data has become an important research direction and application demand. Traditional unimodal processing systems (only processing one type of data, such as text or images) can no longer meet the diverse application needs, while multimodal processing systems (capable of processing multiple types of data at the same time) are valued because they can provide richer information and more accurate interactive feedback.
[0003] Existing multimodal data processing technologies mainly face the following challenges:
[0004] Insufficient information fusion between modalities: Information between multiple modal data (such as text and video) is often difficult to effectively integrate, resulting in the final generated content to be information-isolated and semantically incoherent.
[0005] Data generation consistency issues: When generating text such as image descriptions, it is often difficult to ensure the consistency and mutual complementation of the text and image content, which affects the user experience and the practicality of the system.
[0006] Catastrophic forgetting: Multimodal systems tend to forget old knowledge (catastrophic forgetting) when continuously learning new modal data processing tasks, especially when the model needs to be constantly updated.
[0007] Scarcity and bias of training data: The training samples for a specific modality combination may be extremely limited, which limits the training effect and generalization ability of the model.
[0008] In order to solve these problems, it is necessary to develop and train a new type of multimodal large model that can efficiently understand and generate multimodal data including text, images, videos and audio under a unified framework. This model can not only improve the efficiency of information fusion between modalities, but also improve the consistency and quality of generated data, reduce the problem of catastrophic forgetting, and effectively utilize existing scarce data resources to support complex multimodal application scenarios. Summary of the invention
[0009] In order to solve the technical problems existing in the prior art, the existing multimodal large model cannot efficiently understand and generate multiple modal data including text, images, videos and audios under a unified framework, cannot improve the efficiency of information fusion between modalities, and cannot improve the consistency and quality of generated data and reduce catastrophic forgetting, the technical solution provided by the present invention is:
[0010] A Chinese multimodal data generation method based on a unified framework, comprising:
[0011] The steps of collecting multimodal data and preprocessing them;
[0012] A step of performing feature extraction on the preprocessed multimodal data to obtain a text feature vector and a multimodal feature vector;
[0013] Aligning the row features of the multimodal feature vector, and then concatenating the aligned multimodal features with the text feature vector to obtain a feature embedding representation of the user input data;
[0014] The step of embedding the feature representation into a multimodal macro model to generate multimodal content;
[0015] The step of integrating the generated multimodal content and outputting it in sequence.
[0016] Furthermore, a preferred embodiment is provided, wherein the preprocessing includes image resizing, cropping and normalization processing, video key frame extraction and audio sampling rate conversion.
[0017] Furthermore, a preferred embodiment is provided, in which, in the multimodal large model, the feature embedding representation is comprehensively analyzed through expert weighted scoring in the model to generate multimodal content.
[0018] Furthermore, a preferred embodiment is provided, wherein the multimodal data includes images, videos and audios.
[0019] Furthermore, a preferred implementation is provided, in which the preprocessed multimodal data is input into a multimodal encoding model to obtain a text feature vector and a multimodal feature vector.
[0020] Furthermore, a preferred embodiment is provided, in which the multimodal feature vector rows are feature aligned through a bridging layer model.
[0021] Based on the same inventive concept, the present invention also provides a Chinese multimodal data generation device based on a unified framework, comprising:
[0022] A module for collecting and preprocessing multimodal data;
[0023] A module for extracting features from the preprocessed multimodal data to obtain text feature vectors and multimodal feature vectors;
[0024] Aligning the row features of the multimodal feature vector, and then concatenating the aligned multimodal features with the text feature vector to obtain a module for embedding the feature representation of the user input data;
[0025] Embedding the features into a representation input multimodal macromodel to generate a module for multimodal content;
[0026] A module that integrates the generated multimodal content and outputs it in sequence.
[0027] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes the method described.
[0028] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method described.
[0029] Based on the same inventive concept, the present invention also provides a computer program product, which is a computer program. When the computer program is executed, the method described above is implemented.
[0030] Compared with the prior art, the technical solution provided by the present invention is beneficial in that:
[0031] The present invention provides a Chinese multimodal data generation method based on a unified framework. Through data preprocessing, the format and size of image, video and audio data are unified, making subsequent processing more efficient and accurate. Compared with existing multimodal data processing methods, this preprocessing method significantly improves the compatibility and consistency of data, reduces the interference of data noise and redundant information, and thus improves the performance and stability of the overall system.
[0032] The present invention provides a Chinese multimodal data generation method based on a unified framework. In the feature extraction step, a pre-trained model is used to extract features from text and multimodal data, ensuring the high quality and high dimension of the extracted features. Compared with the traditional manual feature extraction method, the pre-trained model can automatically learn the deep-level features of the data, thereby better capturing the complex relationship between multimodal data and improving the accuracy and expressiveness of feature representation.
[0033] The present invention provides a Chinese multimodal data generation method based on a unified framework. In the feature alignment and splicing steps, the introduction of the bridge layer model realizes the alignment of semantics and feature dimensions, so that data of different modalities can be fused in a unified feature space. This method is more intelligent than the existing simple feature splicing method, can more accurately capture the correlation between multimodal data, and improves the effect of data fusion and the system's understanding ability.
[0034] The present invention provides a Chinese multimodal data generation method based on a unified framework. In the large model processing and generation steps, the expert weighted scoring mechanism is used to comprehensively analyze user instructions and generate multimodal content. This method is more flexible and intelligent than the traditional single model processing method, and can dynamically adjust model parameters according to the needs of different tasks, thereby improving the accuracy and diversity of multimodal content generation.
[0035] The present invention provides a Chinese multimodal data generation method based on a unified framework. In the multimodal content integration and output step, the generated text, image, video, and audio are integrated according to user needs and output in sequence. This method is more user-friendly and practical than the existing independent output method, and can provide a more complete and coherent multimodal information service, thereby improving the user experience and the practical value of the system.
[0036] The present invention provides a Chinese multimodal data generation method based on a unified framework, which is suitable for understanding and generating multimodal data including text, images, videos and audios. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a flowchart of a Chinese multimodal data generation method based on a unified framework;
[0038] Figure 2 This is a model architecture diagram of a multimodal large model in a Chinese multimodal data generation method based on a unified framework;
[0039] Figure 3 This is an example diagram of a Chinese multimodal data generation method based on a unified framework. DETAILED DESCRIPTION
[0040] In order to make the advantages and benefits of the technical solution provided by the present invention clear, the technical solution provided by the present invention is now further described in detail with reference to the accompanying drawings, specifically:
[0041] Implementation 1: This implementation provides a method for generating Chinese multimodal data based on a unified framework, including:
[0042] The steps of collecting multimodal data and preprocessing them;
[0043] A step of performing feature extraction on the preprocessed multimodal data to obtain a text feature vector and a multimodal feature vector;
[0044] Aligning the row features of the multimodal feature vector, and then concatenating the aligned multimodal features with the text feature vector to obtain a feature embedding representation of the user input data;
[0045] The step of embedding the feature representation into a multimodal macro model to generate multimodal content;
[0046] The step of integrating the generated multimodal content and outputting it in sequence.
[0047] Specific:
[0048] Step 1: Multimodal Data Preprocessing
[0049] Input: raw multimodal data (images, videos, audio, etc.)
[0050] Image data processing:
[0051] Use image processing tools to resize, crop, and normalize images.
[0052] Make sure your images are of uniform size and format for easy processing.
[0053] Video data processing:
[0054] Extract key frames or key frame sequences from a video.
[0055] Convert videos to image sequences for processing.
[0056] Audio data processing:
[0057] Perform sampling rate conversion and sound feature extraction.
[0058] Converts audio signals into a digital format that can be processed by analog.
[0059] Output: preprocessed multimodal data, images, videos, and audio data in unified format and size.
[0060] Step 2: Feature Extraction
[0061] Input: Preprocessed multimodal data
[0062] Text feature extraction:
[0063] Use the word segmentation tool to segment the text.
[0064] The word segmentation results are input into a pre-trained word embedding model (such as the ELMO model) to obtain the vector representation of each word segmentation.
[0065] Multimodal feature extraction:
[0066] The preprocessed image, video, and audio data are input into a multimodal coding model (such as the ViT model) to obtain a vector representation of the multimodal data.
[0067] Output: text feature vector and multimodal feature vector.
[0068] Step 3: Multimodal feature alignment and splicing
[0069] Input: text feature vector and multimodal feature vector
[0070] Feature Alignment:
[0071] The multimodal feature vector is input into the bridge layer model (such as a three-layer perceptron model) to achieve alignment of semantics and feature dimensions.
[0072] Feature stitching:
[0073] The aligned multimodal features and text features are concatenated in the order of user input to obtain the feature embedding representation of the user input data.
[0074] Output: Feature embedding representation of user input data.
[0075] Step 4: Large model processing and generation
[0076] Input: Feature embedding representation of user input data
[0077] Large model processing:
[0078] The feature embedding representation is fed into a large multimodal model.
[0079] The user instructions are comprehensively analyzed through weighted scoring by experts in the model.
[0080] Generate multimodal content:
[0081] If it is a question-answering task, generate relevant text answers.
[0082] If it is a generation task, the corresponding diffusion model downstream of the shared multimodal framework is used to generate images, audio, video and other content.
[0083] Output: Generated multimodal content.
[0084] Step 5: Multimodal content integration and output
[0085] Input: Generated multimodal content (text, images, video, audio, etc.)
[0086] Integration content:
[0087] The generated text, images, videos and audio are integrated according to user needs.
[0088] Output:
[0089] The integrated multimodal content is output in sequence to meet user needs.
[0090] Output: integrated multimodal content.
[0091] Implementation method 2. This implementation method is a further limitation of the Chinese multimodal data generation method based on a unified framework provided in implementation method 1. The preprocessing includes image resizing, cropping and normalization, video key frame extraction and audio sampling rate conversion.
[0092] Implementation method three: This implementation method is a further limitation of the Chinese multimodal data generation method based on a unified framework provided in implementation method one. In the multimodal large model, the feature embedding representation is comprehensively analyzed through weighted scoring by experts in the model to generate multimodal content.
[0093] Implementation method 4: This implementation method is a further limitation of the Chinese multimodal data generation method based on a unified framework provided in implementation method 1, wherein the multimodal data includes images, videos and audios.
[0094] Implementation method five: This implementation method is a further limitation of the Chinese multimodal data generation method based on a unified framework provided in implementation method one. The preprocessed multimodal data is input into a multimodal encoding model to obtain a text feature vector and a multimodal feature vector.
[0095] Implementation method six: This implementation method is a further limitation of the Chinese multimodal data generation method based on a unified framework provided in implementation method one, and aligns the multimodal feature vector row features through a bridging layer model.
[0096] Embodiment 7: This embodiment provides a Chinese multimodal data generation device based on a unified framework, including:
[0097] A module for collecting and preprocessing multimodal data;
[0098] A module for extracting features from the preprocessed multimodal data to obtain text feature vectors and multimodal feature vectors;
[0099] Aligning the row features of the multimodal feature vector, and then concatenating the aligned multimodal features with the text feature vector to obtain a module for embedding the feature representation of the user input data;
[0100] Embedding the features into a representation input multimodal macromodel to generate a module for multimodal content;
[0101] A module that integrates the generated multimodal content and outputs it in sequence.
[0102] Embodiment 8: This embodiment provides a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes the method provided in embodiment 1.
[0103] Embodiment 9: This embodiment provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method provided in embodiment 1.
[0104] Embodiment 10: This embodiment provides a computer program product, which is a computer program. When the computer program is executed, the method provided in embodiment 1 is implemented.
[0105] Implementation Method XI: Combination Figure 1-3 This embodiment is described in detail by using specific examples to further describe the technical solution provided above. Specifically:
[0106] Example 1: Combination Figure 1 This embodiment is described to provide a large model technology for understanding and generating Chinese multimodal data based on a unified framework, which specifically includes the following steps:
[0107] Step 1: Process the input multimodal data such as images, videos, and audio and transform them into a format that the model can process.
[0108] In real life, after the acquisition of image, video, audio and other data, the size and format are different, which will cause a sharp decline in effect and efficiency in subsequent processing, so it is necessary to preprocess them into a unified format and size. For example, for image data, preprocessing operations such as scaling, cropping, and normalization can be performed to ensure that the image has a uniform size and format. For video data, key frames or key frame sequences can be extracted and converted into image sequences for processing. For audio data, sampling rate conversion, sound feature extraction and other operations can be performed so that the model can effectively analyze and process it.
[0109] This step is one of the pre-steps. When processing multimodal data, corresponding pre-processing operations are required to ensure that the data format is unified and meets the model input requirements, which can greatly facilitate subsequent data analysis and processing.
[0110] Step 2. Use the pre-trained word embedding model to obtain text features, and use the multimodal encoder to obtain aligned multimodal features such as images, videos, and audio;
[0111] This step is one of the steps to understand Chinese multimodal data. The text obtained in step 1 is segmented and then input into the pre-trained model to obtain a vector representation of each segmented word. At the same time, the images, videos, and audios of the same format and size in step 1 are input into the pre-trained multimodal encoding model to obtain a vector representation of multimodal data.
[0112] Step 3. Align the multimodal features obtained in step 2 with the text features through a bridge layer, and then concatenate them with the text features to obtain the feature embedding representation of the user input data;
[0113] This step is one of the steps to understand Chinese multimodal data. The multimodal data embedding representation obtained in step 2 is input into the model with a three-layer perceptron as a bridge layer to achieve alignment of semantics and feature dimensions. Then, the aligned multimodal feature embedding representation and the text feature embedding representation are concatenated in the order of user input to obtain a unified feature embedding representation of the user input data.
[0114] Step 4: The unified input obtained in step 3 is passed to the multimodal large model, and the experts in the multimodal large model perform weighted scoring to obtain the large model output;
[0115] This step is to understand the input Chinese multimodal data through a large model. The model will integrate information from each modality, and different experts will analyze user instructions. Finally, the scores of all experts will be combined for understanding and generation. If it is a question-and-answer task of the corresponding modality, the model will generate relevant text to answer user questions. If it is a generation task, the model will use the corresponding diffusion model downstream of the shared multimodal framework built on the generation end to generate images, audio, video, etc. according to the instructions.
[0116] Embodiment 2: This embodiment is different from the specific implementation mode 1 in that in step 1, multimodal data such as images, videos, and audios in the input are processed. The specific implementation process includes the following steps:
[0117] Step 1.1 Use the library function in the Python programming language to resize the source image to ensure that the image has a uniform size and format;
[0118] Step 1.2 uses the library function in the Python programming language to extract key frames or key frame sequences from the source video and convert them into image sequences for processing;
[0119] Step 1.3 uses the library functions in the Python programming language to sample and quantize the source audio signal to convert the analog audio signal into digital form;
[0120] Embodiment 3: This embodiment is different from the second embodiment in that the pre-trained word embedding model involved in step 2 obtains text features, and uses a multimodal encoder to embed the images, videos, audio, etc. obtained in step 1. The model structure is as follows: Figure 2 As shown in the figure, the pre-trained model used for text is the ELMO model, which consists of three layers of bidirectional LSTM, and each layer can be used as a separate word embedding representation. The pre-trained model used for multimodal data is the LanguageBind model, which consists of ViT, and each ViT is used as an embedding model of a modality. Specifically, the following steps are included:
[0121] Step 2.1 Use a word segmentation tool to segment the obtained text to obtain a word segmentation result;
[0122] Step 2.2 Input the word segmentation results obtained in step 2.1 into the pre-trained model ELMO to obtain the word embedding representation of each word in each layer Where l∈{0,1,2} represents the first, second, and third LSTM representations of EMLO respectively; i∈[0,n] represents the word segmentation results at each position of the article, and n represents the number of word segmentation results of the article;
[0123] Step 2.3: The three layers of ELMO have different weights. The embedding of each word is obtained according to the weights and the word embedding representation obtained in step 2.2. i , the formula is as follows:
[0124]
[0125] Step 2.4 Input the images, videos, and audios of uniform size and format obtained in steps 1.1, 1.2, and 1.3 into the pre-trained model LanguageBind to obtain the embedding representation of multimodal data. m .
[0126] Embodiment 4: This embodiment is different from the specific implementation mode 3 in that the multimodal features involved in step 3 are aligned with the text features through the bridge layer, and are spliced with the text features to obtain the user input data feature embedding representation. The specific steps are as follows:
[0127] Step 3.1 Embedding the multimodal features obtained in step 2.4 m Input into the bridge layer model MLP,
[0128] Get the embedding representation of each multimodal data and the alignment of the LLM input space semantics and spatial dimensions Embedding' m , the formula is as follows:
[0129] hidden_state1=ReLU(W1·Embedding m +b1),
[0130] hidden_state2=ReLU(W2·hidden_state1+b2),
[0131] Embedding m =W3·hidden_state2+b3;
[0132] Step 3.2: The multimodal embedding representation obtained in step 3.1 and semantically aligned with the LLM input space is represented as Embedding' m Embedding representation of each word in the text i The concatenation is performed to obtain an input embedding representation that is aligned with the four modalities of text, image, video, and audio.
[0133] Embodiment 5: This embodiment is different from the specific implementation mode 4 in that the pre-trained large language model involved in step 4 passes the input embedding results of the four modal alignments of text, image, video, and audio obtained in step 3 to the multimodal large model, and is output after being scored by each expert. The specific steps are as follows:
[0134] Step 4.1: According to the input embedding representation obtained in step 3.2, it is sent to each Transformer block of LLM, and the attention matrix therein is reduced in dimension using LoRA. For the original LoRA model, its function module includes three parts: Query, Key, and Value. Based on this, the present invention uses a hybrid expert network. For each attention matrix, eight experts are used for scoring. Each expert is composed of a LoRA model. The formula is as follows:
[0135]
[0136] Where x represents the embedded representation of the input, α represents the scaling factor, which is a hyperparameter used to measure the magnitude of the change in the original model weight due to dimensionality reduction, and r represents the dimension after dimensionality reduction. The model weight W is decomposed into two matrices, B and A, where
[0137] Step 4.2: Output the weighted expert scoring results obtained in step 4.1 through the routing model:
[0138]
[0139] G(x)=W g x+b g ;
[0140] Where N represents the number of experts, W0x represents the calculation result of the original parameters of the model for the input x, because LoRA uses the original value plus the change value to calculate the final result, G(x) i is a trainable routing network that is used to score the weights of each expert, E i (x) represents each expert network, where the expert network is the attention matrix Query, Key, and Value after LoRA dimensionality reduction. Through such a modular expert system design, the model can activate the corresponding processing module according to the modal combination of the input data, which not only improves the processing efficiency, but also reduces the catastrophic forgetting of the model when facing new tasks.
[0141] Step 4.3 passes the result of step 4.2 until the output result of the last layer is decoded to obtain the output text. The formula is as follows:
[0142] id=argmax(W output o last_layer +b output ).
[0143] Embodiment 6: This embodiment is different from the specific implementation mode 5 in that after the text is generated in step 4, a special mark is used to inform the downstream model to generate images, videos, and audios. The specific steps are as follows:
[0144] Step 4.3: According to the ID output in step 4.3, determine whether it belongs to the special tag we specified: [IMG], [VID], [AUD].
[0145] Step 4.4 Based on the result in step 4.4, if there is a special tag [IMG], the text corresponding to the special tag is input into the Stable Diffusion model; if there is a special tag [VID], the text corresponding to the special tag is input into the ZeroScope model; if there is a special tag [AUD], the text corresponding to the special tag is input into the Audioldm model.
[0146] Step 4.5 integrates the output text as well as the images, videos, and audios and outputs them in sequence.
[0147] In this embodiment, a simple automatic Chinese multimodal data understanding and generation module can be implemented according to the above steps. The module can be embedded in any existing system to achieve a plug-and-play effect. The specific beneficial effects of the invention are as follows:
[0148] This embodiment follows Figure 1The process is carried out to build a large model system for understanding and generating Chinese multimodal data based on a unified framework. After the system implemented in this embodiment is started, the pre-trained model and resources will be loaded first. After that, the system will transform the multimodal data according to the user's input this time, and then the extracted text features will be spliced with the multimodal features and passed through the pre-trained model to obtain a large model input with a unified representation. Finally, the large model text output containing special identifiers will be routed to the downstream diffusion model for multimodal generation to obtain the final output.
[0149] Figure 3 These are some randomly selected examples of multi-round conversations between users and the system, including user input, the system's response to the user, and the system's multimodal generation results.
[0150] Randomly selected user input examples, such as Figure 3 , some common user input scenarios and requirements were selected, for example, users require the generation of multimodal content such as audio and video, and ask questions about scene information in pictures.
[0151] The final actual operation results of the invention can be seen Figure 3 As shown. According to the system response and generation results in the figure, it can be seen that the Chinese multimodal data understanding and generation large model system implemented in this embodiment can not only accurately understand the multimodal content input by the user, but also generate multimodal content according to the specific needs of the user, and attach a detailed text description to the generated multimodal content. And from the examples in the figure, it can be seen that the system implemented in this embodiment supports multi-round dialogues, combined input and output of any modality, and has multi-round context understanding capabilities, with good overall performance and applicability. This embodiment can be used in various multimodal interaction scenarios, improves the ability of Chinese multimodal information processing, and promotes the development of Chinese multimodal technology.
[0152] The technical solution provided by the present invention is further described in detail above through several specific implementation modes in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific implementation modes described above are not intended to be used as limitations on the present invention. Any reasonable modification and improvement of the present invention, combination of implementation modes and equivalent substitution within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A Chinese multimodal data generation method based on a unified framework, characterized in that: include: The steps of collecting multimodal data and preprocessing them; A step of performing feature extraction on the preprocessed multimodal data to obtain a text feature vector and a multimodal feature vector; Aligning the row features of the multimodal feature vector, and then concatenating the aligned multimodal features with the text feature vector to obtain a feature embedding representation of the user input data; The step of embedding the feature representation into a multimodal macro model to generate multimodal content; The step of integrating the generated multimodal content and outputting it in sequence.
2. According to the method for generating Chinese multimodal data based on a unified framework according to claim 1, it is characterized in that: The preprocessing includes image resizing, cropping and normalization, video key frame extraction and audio sampling rate conversion.
3. According to the method for generating Chinese multimodal data based on a unified framework according to claim 1, it is characterized in that: In the multimodal large model, the feature embedding representation is comprehensively analyzed through expert weighted scoring in the model to generate multimodal content.
4. According to the method for generating Chinese multimodal data based on a unified framework according to claim 1, it is characterized in that: The multimodal data includes images, videos and audios.
5. The Chinese multimodal data generation method based on a unified framework according to claim 1 is characterized in that: The preprocessed multimodal data is input into a multimodal encoding model to obtain a text feature vector and a multimodal feature vector.
6. The Chinese multimodal data generation method based on a unified framework according to claim 1 is characterized in that: The multimodal feature vector rows are feature aligned through a bridge layer model.
7. A Chinese multimodal data generation device based on a unified framework, characterized in that: include: A module for collecting and preprocessing multimodal data; A module for extracting features from the preprocessed multimodal data to obtain text feature vectors and multimodal feature vectors; Aligning the row features of the multimodal feature vector, and then concatenating the aligned multimodal features with the text feature vector to obtain a module for embedding the feature representation of the user input data; Embedding the features into a representation input multimodal macromodel to generate a module for multimodal content; A module that integrates the generated multimodal content and outputs it in sequence.
8. A computer storage medium for storing a computer program, characterized in that: When the computer program is read by a computer, the computer executes the method of claim 1 .
9. A computer, comprising a processor and a storage medium, characterized in that: When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1 .
10. A computer program product, being a computer program, characterized in that When the computer program is executed, the method of claim 1 is implemented.