Multimodal interaction-oriented large language model joint training method and system

Abstract

The application provides a large language model joint training method and system for multi-modal interaction, the method comprising the following steps: obtaining an original video containing subtitles and sound, and disassembling the original video into multiple data groups, wherein each data group comprises corresponding text data, voice data and image data; separately extracting the data of each mode in the multiple data groups to obtain first training data composed of multiple text data, second training data composed of multiple voice data and third training data composed of multiple image data; in a first training stage, training a large language model through the first training data, training the large language model through the second training data and training the large language model through the third training data; in a second training stage, taking the multiple data groups as fourth training data, and training the large language model through the fourth training data.

Description

Technical Field

[0001] This invention relates to the field of model training technology, specifically to a joint training method and system for large language models oriented towards multimodal interaction. Background Technology

[0002] With the increasing prevalence of large language models, people have higher and higher requirements for their intelligence. For example, in scenarios involving interaction with virtual humans, users expect machines to be as intelligent as real people, correctly understand the user's actual intentions, and provide appropriate feedback.

[0003] However, current virtual human technology can only understand user intent based on text or text generated from speech recognition, and cannot provide actions and expressions that closely match the response content in real interaction scenarios. For example, when a virtual human speaks, it always uses fixed gestures and expressions, which leads to inaccurate understanding of the interaction intent and inappropriate and unnatural responses.

[0004] Therefore, how to enable large language models to understand multiple modal information of users, including speech, text, and visuals, and to output such information, thereby achieving true multimodal interaction, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a joint training method and system for large language models oriented towards multimodal interaction. This method effectively improves the understanding and generation capabilities of large language models for multiple modal information, including text, speech, and images. It is also suitable for multimodal interaction scenarios and more closely resembles natural and real-world human-to-human interactions.

[0006] The technical solution adopted in this invention is as follows:

[0007] A joint training method for a large language model oriented towards multimodal interaction includes the following steps: acquiring an original video containing subtitles and audio, and decomposing the original video into multiple data groups, wherein each data group includes corresponding text data, speech data, and image data; extracting the data of each modality from the multiple data groups separately to obtain first training data consisting of multiple text data, second training data consisting of multiple speech data, and third training data consisting of multiple image data; in a first training stage, training the large language model using the first training data, training the large language model using the second training data, and training the large language model using the third training data, respectively; in a second training stage, using the multiple data groups as fourth training data, and training the large language model using the fourth training data.

[0008] The large language model includes a first encoder, a second encoder, a third encoder, a cross-attention layer, a feature decomposition layer, a first decoder, a second decoder, and a third decoder. The first encoder, the second encoder, and the third encoder are respectively connected to the cross-attention layer. The first decoder, the second decoder, and the third decoder are respectively connected to the feature decomposition layer. The feature decomposition layer is connected to the cross-attention layer.

[0009] The first training data, the second training data, and the third training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The text data, speech data, and image data in the fourth training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The first decoder, the second decoder, and the third decoder respectively decode the text features, speech features, and image features.

[0010] The first encoder, the second encoder, and the third encoder are respectively a Transformer encoder, a CNN encoder, and a ViT (Vision Transformer) encoder, and the first decoder, the second decoder, and the third decoder are respectively a Transformer decoder, a WaveNet decoder, and a diffusion model.

[0011] Each text data in the first training data serves as a training sample data, and at least one token in the text data is masked; each voice data in the second training data serves as a training sample data, and at least one phone in the voice data is masked; each image data in the third training data serves as a training sample data, and at least one patch in the image data is masked.

[0012] Each data group in the fourth training data serves as a training sample data, and one or two random modalities in the data group are masked.

[0013] A joint training system for a large language model oriented towards multimodal interaction includes: an acquisition module for acquiring an original video containing subtitles and audio, and decomposing the original video into multiple data groups, wherein each data group includes corresponding text data, audio data, and image data; an extraction module for extracting the data of each modality from the multiple data groups separately to obtain first training data consisting of multiple text data, second training data consisting of multiple audio data, and third training data consisting of multiple image data; and a training module for training the large language model using the first training data, the second training data, and the third training data respectively in a first training phase, and using the multiple data groups as fourth training data in a second training phase to train the large language model.

[0014] The large language model includes a first encoder, a second encoder, a third encoder, a cross-attention layer, a feature decomposition layer, a first decoder, a second decoder, and a third decoder. The first encoder, the second encoder, and the third encoder are respectively connected to the cross-attention layer. The first decoder, the second decoder, and the third decoder are respectively connected to the feature decomposition layer. The feature decomposition layer is connected to the cross-attention layer.

[0015] The first training data, the second training data, and the third training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The text data, speech data, and image data in the fourth training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The first decoder, the second decoder, and the third decoder respectively decode the text features, speech features, and image features.

[0016] The first encoder, the second encoder, and the third encoder are respectively a Transformer encoder, a CNN encoder, and a ViT encoder, and the first decoder, the second decoder, and the third decoder are respectively a Transformer decoder, a WaveNet decoder, and a diffusion model.

[0017] Each text data in the first training data serves as a training sample data, and at least one token in the text data is masked; each voice data in the second training data serves as a training sample data, and at least one phone in the voice data is masked; each image data in the third training data serves as a training sample data, and at least one patch in the image data is masked.

[0018] Each data group in the fourth training data serves as a training sample data, and one or two random modalities in the data group are masked.

[0019] The beneficial effects of this invention are:

[0020] This invention extracts a data set containing multimodal data from the original video containing subtitles and audio, and uses this data set to train a large language model in two stages. In the first stage, the model is trained separately with each unimodal data set, and in the second stage, it is trained with multimodal data. This effectively improves the large language model's ability to understand and generate multiple modal information such as text, speech, and images, and is suitable for multimodal interaction scenarios, making it closer to natural and real-world human-to-human interactions. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for joint training of large language models for multimodal interaction according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of a large language model according to an embodiment of the present invention;

[0023] Figure 3 This is a block diagram of a large language model joint training system for multimodal interaction according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figure 1 As shown in the figure, the joint training method for large language models oriented towards multimodal interaction according to an embodiment of the present invention includes the following steps:

[0026] S1: Obtain the original video containing subtitles and audio, and decompose the original video into multiple data groups, where each data group includes corresponding text data, audio data and image data.

[0027] In one embodiment of the present invention, the original video may be an interview program video, news interview video, etc. containing subtitles and dialogue audio, wherein the subtitles are text corresponding to the dialogue content.

[0028] In one embodiment of the present invention, the audio and video can be segmented using the switching time points of each subtitle as boundaries, resulting in one subtitle, one audio segment, and one video segment within each time period. The text content of one subtitle within a time period is considered as text data, the audio segment within that time period is considered as corresponding audio data, and an image frame from the video segment within that time period is considered as corresponding image data. The image frame can be an image frame from the middle of the video segment or an image frame randomly selected from the video segment. Since the original video is an interview program video, news interview video, etc., the image data in each data group consists of images of people, containing information such as people's actions and / or expressions.

[0029] The larger the number of data sets acquired in this step, the better the subsequent training effect will be, since it is used to train a large language model. However, the computing power of the processor running the model should also be considered.

[0030] S2, extract the data of each modality from the multiple data sets separately to obtain the first training data consisting of multiple text data, the second training data consisting of multiple speech data, and the third training data consisting of multiple image data.

[0031] S3, in the first training phase, trains the large language model using the first training data, the second training data, and the third training data, respectively.

[0032] S4, in the second training phase, uses multiple data sets as the fourth training data to train the large language model.

[0033] The large language model of this invention can be applied to virtual humans displayed on a screen, and is suitable for virtual human interaction scenarios such as games, intelligent customer service, and questionnaires. In one embodiment of this invention, such as Figure 2As shown, the large language model includes a first encoder, a second encoder, a third encoder, a cross-attention layer, a feature decomposition layer, a first decoder, a second decoder, and a third decoder. The first encoder, the second encoder, and the third encoder are connected to the cross-attention layer, the first decoder, the second decoder, and the third decoder are connected to the feature decomposition layer, and the feature decomposition layer is connected to the cross-attention layer.

[0034] The first training data, the second training data, and the third training data are encoded by the first encoder, the second encoder, and the third encoder, respectively. The text data, speech data, and image data in the fourth training data are encoded by the first encoder, the second encoder, and the third encoder, respectively. The first decoder, the second decoder, and the third decoder decode the text features, speech features, and image features, respectively.

[0035] In the second training phase, when the large language model simultaneously inputs data from multiple modalities, after the first to third encoders encode the data of the corresponding modalities, the cross-attention layer can calculate the attention weights between features of different modalities to achieve cross-modal feature fusion; the feature decomposition layer can extract relevant features from the fused features based on the query vectors of each modality through attention decomposition, and output them to the decoders corresponding to each modality for decoding.

[0036] In the first training phase, when the large language model only inputs data of a single modality at the same time, the query, key, and value of the cross-attention layer are unified into features of the same modality, that is, it degenerates into a self-attention layer, thereby realizing the processing of single-modality features; correspondingly, the feature decomposition layer can directly transmit the features to the corresponding decoder for decoding.

[0037] In one specific embodiment of the present invention, the first encoder is a Transformer encoder, the second encoder is a CNN, and the third encoder is a ViT encoder; the first decoder is a Transformer decoder, the second decoder is a WaveNet, and the third decoder is a diffusion model.

[0038] In one embodiment of the present invention, each text data in the first training data is used as a training sample data, and at least one token in the text data is masked. Here, a token refers to a basic unit of text data when processed by the model; this unit can be a word, a sub-word, a character, etc.

[0039] Each speech data point in the second training data serves as a training sample, and at least one phoneme in the speech data is masked. Here, "phoneme" refers to a phoneme.

[0040] Each image in the third training data serves as a training sample, and at least one patch in the image data is masked. Here, a patch refers to an image block.

[0041] The training in the first training phase of step S3 can improve the large language model's ability to understand and generate data for each modality.

[0042] In one embodiment of the present invention, each data group in the fourth training data serves as a training sample data, and one or two modalities in the data group are randomly masked. The masking of text data, speech data, and image data may respectively refer to masking all tokens of text, setting all amplitudes or intensity of the waveform data or spectrogram of speech to 0, and setting all pixel values ​​of images to 0.

[0043] The second training phase is conducted after the first training phase. Through the training in the second training phase of step S4, the ability of the large language model to understand the correspondence between various modalities can be improved, and the cross-modal generation ability can be enhanced. Furthermore, with the training foundation of the first training phase, and based on the fact that the large language model already has the ability to understand and generate single modalities, the training effect of cross-modal ability is even better.

[0044] According to embodiments of the present invention, the large language model joint training method for multimodal interaction decomposes a data set containing multimodal data from the original video containing subtitles and audio, and uses this data set to train the large language model in two stages. In the first stage, the model is trained separately with each single-modal data, and in the second stage, it is trained with multimodal data. This effectively improves the large language model's ability to understand and generate multiple modal information such as text, speech, and images, and is suitable for multimodal interaction scenarios, more closely resembling natural and real-world human-to-human interactions.

[0045] Corresponding to the above embodiments of the joint training method for large language models oriented towards multimodal interaction, the present invention also proposes a joint training system for large language models oriented towards multimodal interaction.

[0046] like Figure 3As shown, the large language model joint training system for multimodal interaction according to an embodiment of the present invention includes: an acquisition module 10, an extraction module 20, and a training module 30. The acquisition module 10 is used to acquire the original video containing subtitles and audio, and decompose the original video into multiple data groups, wherein each data group includes corresponding text data, audio data, and image data; the extraction module 20 is used to extract the data of each modality from the multiple data groups separately, obtaining first training data composed of multiple text data, second training data composed of multiple audio data, and third training data composed of multiple image data; the training module 30 is used to train the large language model using the first training data, the second training data, and the third training data respectively in the first training stage, and to train the large language model using the multiple data groups as fourth training data in the second training stage.

[0047] In one embodiment of the present invention, the original video may be an interview program video, news interview video, etc. containing subtitles and dialogue audio, wherein the subtitles are text corresponding to the dialogue content.

[0048] In one embodiment of the present invention, the acquisition module 10 can segment the audio and video based on the switching time points of each subtitle, obtaining one subtitle, one audio segment, and one video segment within each time period. The text content of one subtitle within a time period is taken as text data, the audio segment within that time period is taken as the corresponding audio data, and an image frame from the video segment within that time period is taken as the corresponding image data. The image frame can be an image frame at the middle time point of the video segment, or an image frame randomly selected from the video segment. Since the original video is an interview program video, news interview video, etc., the image data in each data group is an image of a person, which includes information such as the person's actions and / or expressions.

[0049] The number of data sets acquired by module 10 is important because it is used to train a large language model. The larger the number, the better the subsequent training effect. However, it should also be considered in conjunction with the computing power of the processor running the model.

[0050] The large language model of this invention can be applied to virtual humans displayed on a screen, and is suitable for virtual human interaction scenarios such as games, intelligent customer service, and questionnaires. In one embodiment of this invention, such as Figure 2 As shown, the large language model includes a first encoder, a second encoder, a third encoder, a cross-attention layer, a feature decomposition layer, a first decoder, a second decoder, and a third decoder. The first encoder, the second encoder, and the third encoder are connected to the cross-attention layer, the first decoder, the second decoder, and the third decoder are connected to the feature decomposition layer, and the feature decomposition layer is connected to the cross-attention layer.

[0051] The first training data, the second training data, and the third training data are encoded by the first encoder, the second encoder, and the third encoder, respectively. The text data, speech data, and image data in the fourth training data are encoded by the first encoder, the second encoder, and the third encoder, respectively. The first decoder, the second decoder, and the third decoder decode the text features, speech features, and image features, respectively.

[0052] In the second training phase, when the large language model simultaneously inputs data from multiple modalities, after the first to third encoders encode the data of the corresponding modalities, the cross-attention layer can calculate the attention weights between features of different modalities to achieve cross-modal feature fusion; the feature decomposition layer can extract relevant features from the fused features based on the query vectors of each modality through attention decomposition, and output them to the decoders corresponding to each modality for decoding.

[0053] In the first training phase, when the large language model only inputs data of a single modality at the same time, the query, key, and value of the cross-attention layer are unified into features of the same modality, that is, it degenerates into a self-attention layer, thereby realizing the processing of single-modality features; correspondingly, the feature decomposition layer can directly transmit the features to the corresponding decoder for decoding.

[0054] In one specific embodiment of the present invention, the first encoder is a Transformer encoder, the second encoder is a CNN, and the third encoder is a ViT encoder; the first decoder is a Transformer decoder, the second decoder is a WaveNet, and the third decoder is a diffusion model.

[0055] In one embodiment of the present invention, each text data in the first training data is used as a training sample data, and at least one token in the text data is masked. Here, a token refers to a basic unit of text data when processed by the model; this unit can be a word, a sub-word, a character, etc.

[0056] Each speech data point in the second training data serves as a training sample, and at least one phoneme in the speech data is masked. Here, "phoneme" refers to a phoneme.

[0057] Each image in the third training data serves as a training sample, and at least one patch in the image data is masked. Here, a patch refers to an image block.

[0058] The training in the first phase can improve the large language model's ability to understand and generate data for each modality.

[0059] In one embodiment of the present invention, each data group in the fourth training data serves as a training sample data, and one or two modalities in the data group are randomly masked. The masking of text data, speech data, and image data may respectively refer to masking all tokens of text, setting all amplitudes or intensity of the waveform data or spectrogram of speech to 0, and setting all pixel values ​​of images to 0.

[0060] The second training phase is conducted after the first training phase. Through training in the second training phase, the large language model's ability to understand the correspondence between various modalities can be improved, as can its cross-modal generation ability. Furthermore, with the training foundation from the first training phase, and given that the large language model already possesses the ability to understand and generate single modalities, the training effect on cross-modal capabilities is even better.

[0061] According to an embodiment of the present invention, a large language model joint training system for multimodal interaction is used to decompose a data set containing multimodal data from an original video containing subtitles and audio, and to train the large language model in two stages using the data set. In the first stage, the model is trained separately with each single-modal data, and in the second stage, it is trained with multimodal data. This effectively improves the large language model's ability to understand and generate multiple modal information such as text, speech and images, and is suitable for multimodal interaction scenarios, more closely resembling natural and real-world human-to-human interactions.

[0062] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0063] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0064] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0065] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0066] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0067] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0068] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0069] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0070] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0071] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A joint training method for large language models oriented towards multimodal interaction, characterized in that, Includes the following steps: The original video containing subtitles and audio is acquired, and the original video is decomposed into multiple data groups, wherein each data group includes corresponding text data, audio data and image data; The data of each modality in the multiple data groups are extracted separately to obtain the first training data consisting of multiple text data, the second training data consisting of multiple speech data, and the third training data consisting of multiple image data. In the first training phase, the large language model is trained using the first training data, the second training data, and the third training data, respectively. In the second training phase, the multiple data sets are used as the fourth training data, and the large language model is trained using the fourth training data.

2. The method for joint training of large language models for multimodal interaction according to claim 1, characterized in that, The large language model includes a first encoder, a second encoder, a third encoder, a cross-attention layer, a feature decomposition layer, a first decoder, a second decoder, and a third decoder. The first encoder, the second encoder, and the third encoder are each connected to the cross-attention layer. The first decoder, the second decoder, and the third decoder are each connected to the feature decomposition layer. The feature decomposition layer is connected to the cross-attention layer. The first training data, the second training data, and the third training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The text data, speech data, and image data in the fourth training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The first decoder, the second decoder, and the third decoder respectively decode the text features, speech features, and image features.

3. The method for joint training of large language models for multimodal interaction according to claim 2, characterized in that, The first encoder, the second encoder, and the third encoder are respectively a Transformer encoder, a CNN encoder, and a ViT encoder, and the first decoder, the second decoder, and the third decoder are respectively a Transformer decoder, a WaveNet decoder, and a diffusion model.

4. The method for joint training of large language models for multimodal interaction according to any one of claims 1-3, characterized in that, Each text data in the first training data is used as a training sample data, and at least one token in the text data is masked; Each voice data in the second training data serves as a training sample data, and at least one phone in the voice data is masked; Each image data in the third training data serves as a training sample data, and at least one patch in the image data is masked.

5. The method for joint training of large language models for multimodal interaction according to any one of claims 1-3, characterized in that, Each data group in the fourth training data serves as a training sample data, and one or two random modalities in the data group are masked.

6. A joint training system for large language models oriented towards multimodal interaction, characterized in that, include: The acquisition module is used to acquire the original video containing subtitles and audio, and to decompose the original video into multiple data groups, wherein each data group includes corresponding text data, audio data and image data; An extraction module is used to extract the data of each modality from the multiple data groups separately to obtain first training data consisting of multiple text data, second training data consisting of multiple speech data, and third training data consisting of multiple image data. The training module is configured to train the large language model using the first training data, the second training data, and the third training data respectively in a first training phase, and to train the large language model using the multiple data sets as fourth training data in a second training phase.

7. The joint training system for large language models oriented towards multimodal interaction according to claim 6, characterized in that, The large language model includes a first encoder, a second encoder, a third encoder, a cross-attention layer, a feature decomposition layer, a first decoder, a second decoder, and a third decoder. The first encoder, the second encoder, and the third encoder are each connected to the cross-attention layer. The first decoder, the second decoder, and the third decoder are each connected to the feature decomposition layer. The feature decomposition layer is connected to the cross-attention layer. The first training data, the second training data, and the third training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The text data, speech data, and image data in the fourth training data are respectively encoded by the first encoder, the second encoder, and the third encoder. The first decoder, the second decoder, and the third decoder respectively decode the text features, speech features, and image features.

8. The joint training system for large language models oriented towards multimodal interaction according to claim 7, characterized in that, The first encoder, the second encoder, and the third encoder are respectively a Transformer encoder, a CNN encoder, and a ViT encoder, and the first decoder, the second decoder, and the third decoder are respectively a Transformer decoder, a WaveNet decoder, and a diffusion model.

9. The joint training system for large language models oriented towards multimodal interaction according to any one of claims 6-8, characterized in that, Each text data in the first training data is used as a training sample data, and at least one token in the text data is masked; Each voice data in the second training data serves as a training sample data, and at least one phone in the voice data is masked; Each image data in the third training data serves as a training sample data, and at least one patch in the image data is masked.

10. The joint training system for large language models oriented towards multimodal interaction according to any one of claims 6-8, characterized in that, Each data group in the fourth training data serves as a training sample data, and one or two random modalities in the data group are masked.

Images