A training method, device, equipment and medium for a large full-modal model

Abstract

The present application discloses a training method, device, equipment and medium for an omnimodal large model, and relates to the field of model training. The method includes: using an experience replay pool to store sample data of old tasks; wherein the old tasks are tasks that the omnimodal large model has learned, and the sample data includes data of multiple modalities; when the omnimodal large model learns a new task, the omnimodal large model is trained for multiple rounds based on the sample data of the new task and the sample data of the old task stored in the experience replay pool, and during the training process, the ratio of the sample data of the new task to the sample data of the old task is dynamically adjusted based on the accuracy of the omnimodal large model on the sample data of the old task, so as to obtain the omnimodal large model after joint training. The present application solves the problem of catastrophic forgetting of the omnimodal large model, improves the long-term memory capacity of the omnimodal large model, and ensures that it maintains high performance in the process of continuously learning new tasks.

Description

Technical Field

[0001] The present application relates to the field of model training, and in particular to a training method, device, equipment and medium for a large full-modal model. Background Art

[0002] In recent years, many omnimodal large models have emerged in the field of artificial intelligence, which are able to process multiple types of data, such as text, images, videos, and sounds. Although omnimodal large models perform well on many tasks, they often face the problem of catastrophic forgetting during continuous learning, that is, when omnimodal large models learn new tasks, they tend to forget the previously learned knowledge, resulting in performance degradation.

[0003] Catastrophic forgetting may be caused by factors such as capacity limitations of large omnimodal models, gradient vanishing or exploding problems, and instability in the learning process, which make it difficult for large omnimodal models to maintain long-term memory of old knowledge while continuously learning new tasks.

[0004] The training method of the omnimodal large model directly inputs a large amount of data of different dimensions into the omnimodal large model. As new data is continuously introduced, the omnimodal large model tends to update its parameters to better adapt to new tasks, but this may overwrite or weaken the memory of old tasks. This forgetting phenomenon is particularly serious in early training tasks, because the knowledge learned in these tasks is more easily eroded by subsequent learning processes, resulting in the early large model instruction training tasks being forgotten.

[0005] The omnimodal large model not only faces the challenge of balancing the new and old tasks during the learning process, but also needs to deal with the complexity of information interaction between modalities. For example, if the omnimodal large model ignores the relevance of the text modality of the old task when learning the image modality data of the new task, it may cause the performance of the omnimodal large model on the text modality to degrade.

[0006] Therefore, the catastrophic forgetting problem of large omnimodal models will directly affect the performance stability of the model on multimodal tasks and its cross-modal collaborative learning capabilities. Summary of the invention

[0007] The purpose of this application is to provide a training method, device, equipment and medium for an omnimodal large model, which can solve the catastrophic forgetting problem of the omnimodal large model, improve the long-term memory capacity of the omnimodal large model, and ensure that it maintains high performance in the process of continuously learning new tasks.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] In a first aspect, the present application provides a method for training a large omnimodal model, comprising:

[0010] An experience replay pool is used to store sample data of old tasks; wherein the old tasks are tasks that have been learned by the full-modal large model, and the sample data includes data of multiple modalities;

[0011] When the omnimodal large model learns a new task, the omnimodal large model is trained for multiple rounds based on the sample data of the new task and the sample data of the old task stored in the experience replay pool. During the training process, the ratio of the sample data of the new task to the sample data of the old task is dynamically adjusted according to the accuracy of the omnimodal large model on the sample data of the old task to obtain the omnimodal large model after joint training.

[0012] In a second aspect, the present application provides a training device for a large omnimodal model, comprising:

[0013] An old data storage module is used to use an experience replay pool to store sample data of old tasks; wherein the old tasks are tasks that have been learned by the full-modal large model, and the sample data includes data of multiple modalities;

[0014] The joint training module is used to perform multiple rounds of training on the omnimodal large model when the omnimodal large model learns a new task based on the sample data of the new task and the sample data of the old task stored in the experience replay pool, and dynamically adjust the ratio of the sample data of the new task to the sample data of the old task according to the accuracy of the omnimodal large model on the sample data of the old task during the training process to obtain the omnimodal large model after joint training.

[0015] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned training method for the omnimodal large model.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned training method for the full-modal large model.

[0017] According to the specific embodiments provided in this application, this application has the following technical effects:

[0018] The present application provides a training method, apparatus, equipment and medium for an omnimodal large model. By replaying sample data of old tasks when learning new tasks, the omnimodal large model can retain the knowledge of old tasks, thereby significantly reducing the catastrophic forgetting problem caused by new task training, ensuring that the omnimodal large model does not lose previously learned multimodal information while processing new tasks, improving the long-term memory capacity of the omnimodal large model, and ensuring that it maintains high performance in the process of continuously learning new tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is an application environment diagram of a training method for a full-modal large model in one embodiment of the present application.

[0021] Figure 2 A flowchart of a method for training a large omnimodal model is provided in accordance with an embodiment of the present application.

[0022] Figure 3 Schematic diagram of an experience replay pool in one embodiment of the present application.

[0023] Figure 4 This is a diagram of the complete training process when a full-modal large model learns a new task in one embodiment of the present application.

[0024] Figure 5 A schematic diagram of the functional modules of a full-modal large model training device provided in one embodiment of the present application.

[0025] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] The training method of the full-modal large model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send sample data of the new task to the server 104. After the server 104 receives the sample data of the new task, it performs multiple rounds of training on the full-modal large model according to the sample data of the new task and the sample data of the old task stored in the experience replay pool, and dynamically adjusts the ratio of the sample data of the new task to the sample data of the old task according to the accuracy of the full-modal large model on the sample data of the old task during the training process, so as to obtain the full-modal large model after joint training. The server 104 can feedback the full-modal large model after joint training to the terminal 102. In addition, in some embodiments, the training method of the full-modal large model can also be implemented by the server 104 or the terminal 102 alone.

[0029] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0030] In an exemplary embodiment, Figure 2 As shown, a training method for a full-modal large model is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 202.

[0031] Step 201: Use an experience replay pool to store sample data of old tasks, wherein the old tasks are tasks that have been learned by the full-modality large model, and the sample data include data of multiple modalities.

[0032] Assume existence There are multiple modal data (such as text, image, audio, etc.), and the dataset is recorded as , No. i The sample data of the new task of the modality is recorded as . Set the proportion of sample data of old tasks to α , that is, the proportion of each mode is α , the ratio of sample data for the new task is 1- α Initial value αSet to 0.3.

[0033] Set up a multi-level experience replay pool M , store the sample data of old tasks of each modality in layers, and record For the i Sample data of old tasks of different modalities.

[0034] In an exemplary embodiment, an experience replay pool is used to store data of each modality in layers, and in each layer, the data is grouped and stored according to the relevance of old tasks.

[0035] Specifically, in the experience replay pool, the experiences are stratified according to the modality type and grouped according to the task relevance. Figure 3 As shown in the figure, the sample data of all old tasks are first layered according to the modality (such as text, voice, image, etc.), and each layer stores the sample data of one modality. Then, in each layer, the sample data of this modality is grouped according to the relevance of the old tasks. The layered and grouped storage mechanism ensures that suitable old task samples can be selected when processing new tasks, so as to improve the knowledge retention ability of the full-modality large model between different modalities and effectively deal with the problem of catastrophic forgetting.

[0036] The full-modal large model needs to learn the first task in advance. At this time, only the sample data of the first task needs to be used for training, without considering the old tasks. The loss function is: ;in, is the original loss value, I is the mode number, For the i The first task is the new task at this time.

[0037] Step 202, when the omnimodal large model learns a new task, the omnimodal large model is trained for multiple rounds based on the sample data of the new task and the sample data of the old task stored in the experience replay pool, and during the training process, the ratio of the sample data of the new task to the sample data of the old task is dynamically adjusted according to the accuracy of the omnimodal large model on the sample data of the old task to obtain the omnimodal large model after joint training.

[0038] When training a new task, continuous learning is achieved by mixing sample data of the new task with sample data of the old task in the experience replay pool. When loading sample data of the old task into the experience replay pool, the full-modal large model is dynamically adjusted based on its performance on the old task. α , ensuring that the omnimodal large model effectively retains the knowledge of old tasks to reduce the impact of catastrophic forgetting. Specifically, the performance of the omnimodal large model on old tasks can be measured by testing the accuracy of old tasks. By tracking the performance of the omnimodal large model on sample data of old tasks, dynamic adjustmentα When the performance on the old task is poor, more replay of the old task data is given; when the performance on the old task is good, the replay ratio of the old task is reduced.

[0039] In an exemplary embodiment, the formula is used to determine the proportion of the sample data of the old task in the t -th round of training; where is the proportion of the sample data of the old task in the t -th round of training, is the proportion of the sample data of the old task at the beginning, is the accuracy rate of the full-modal large model on the sample data of the old task in the t -th round of training, is the standard accuracy rate of the full-modal large model on the sample data of the old task before learning the new task. Therefore, if the accuracy rate is low during the training of the new task, will increase, and more sample data of the old task will be loaded; if the accuracy rate is high, will decrease.

[0040] The proportion of the sample data of the new task in the t -th round of training is .

[0041] During each round of training, data is sampled from the experience replay pool and the samples of the new task according to the dynamically adjusted ratio. Assuming the sample batch size is B, the number of samples of the old task and the new task in each round of training are α ×B and (1 - α )×B respectively.

[0042] To balance the old and new task data, the loss function during multiple rounds of training of the full-modal large model is: ; where is the combined loss value, is the loss of the old task for the i -th modality, is the loss of the new task for the i -th modality, is the proportion of the sample data of the old task.

[0043] The above loss function formula can flexibly allocate the weights of the old and new task data in the loss according to the dynamically adjusted ratio, ensuring the continuous learning ability of the full-modal large model.

[0044] To further improve the performance of the full-modal large model, the training method of the full-modal large model further includes the following steps 203 to 204.

[0045] Step 203 : Selecting sample data of old tasks whose correlation with the new task is greater than a set threshold from the experience replay pool as old task samples.

[0046] Randomly draw from the experience replay pool α ×B old task sample data, and by calculating the characteristic KL divergence of the new task sample data and the old task sample data, give priority to the sample data with high modality relevance to the new task to enhance the collaborative learning effect between modalities. Randomly extract (1- α )×B samples, mix the extracted old task samples and new task samples to form a complete training batch, and use these samples to train the full-modal large model.

[0047] Step 204, performing multiple rounds of training on the jointly trained omnimodal large model according to the old task samples and the sample data of the new task to obtain a trained omnimodal large model.

[0048] The loss function for multiple rounds of training of the joint-trained full-modal large model is: ;in, is the dependent loss value, For the i The loss of old task samples of this modality, For the i Examples of old tasks of different modalities.

[0049] Through sample selection driven by inter-modal correlation, the balanced memory between new and old tasks of the omnimodal large model is further optimized.

[0050] like Figure 4 As shown, when the full-modal large model learns a new task, the application first sets up a multi-level experience replay pool to store the sample data of the old task, and initializes the distribution ratio of the new task data to the old task data. Then, in the new training batch, the distribution ratio of the old task data is dynamically adjusted, and samples are taken from the experience replay pool and the sample data of the new task, using the joint loss value Perform preliminary training on the full-modal large model to ensure the overall balance of new and old tasks. Then, when the cross-modal learning effect needs to be further improved, perform cross-modal new task training, select samples from the experience replay pool based on the correlation between modalities, and use the dependent loss value The omnimodal large model is further trained to optimize the performance and efficiency of the omnimodal large model.

[0051] This application can enhance the adaptability of the omnimodal large model in dealing with the catastrophic forgetting problem by introducing a hierarchical experience replay pool, dynamically adjusting the distribution ratio of new task data and old task data, and a sample selection mechanism driven by modal correlation. It can help the omnimodal large model to better retain the knowledge of old tasks and the interaction between modalities during the training of new tasks, thereby improving the comprehensive performance of the omnimodal large model on tasks of different modalities.

[0052] In summary, the present application has at least the following beneficial effects.

[0053] (1) Alleviate catastrophic forgetting. By replaying the data of old tasks when training new tasks, the omnimodal large model can retain the knowledge of old tasks, thereby significantly reducing the catastrophic forgetting problem caused by training new tasks, ensuring that the omnimodal large model does not lose the multimodal information learned previously when processing new tasks.

[0054] (2) Balance the performance of new and old tasks. According to the performance of the omnimodal large model on new and old tasks, dynamically adjust the distribution ratio of new task data and old task data to ensure that the omnimodal large model maintains a good balance between new and old tasks, so that the omnimodal large model can achieve better performance on new tasks while maintaining effective memory of old tasks.

[0055] (3) Flexibility and adaptability. It can be adjusted according to the data characteristics and task requirements of different modalities. The experience replay pool and data mixing strategy can adapt to the data distribution and task complexity of different modalities, thereby improving the adaptability of the full-modal large model.

[0056] The present application also provides an application scenario, which applies the above-mentioned trained full-modal large model. Specifically: the data of multiple modalities are speech, image and text. The new task and the old task both include speech recognition tasks, image classification tasks and text generation tasks. The full-modal large model is a multimodal intelligent question and answer system. The multimodal intelligent question and answer system is used to receive speech, images and text, and perform speech recognition tasks according to speech, convert speech into text, perform image classification tasks according to images, classify images, perform text generation tasks according to text and image classification results, and generate natural language answers.

[0057] In a specific example, a multimodal intelligent question-answering system is an artificial intelligence system that can provide accurate answers based on user input (voice, image, text, etc.), and is commonly used in fields such as online customer service, smart home assistants, and medical consultation.

[0058] In the medical consultation application scenario, the user inputs the question by voice: "What is the medicine in this image?" and provides images of various medicines. The multimodal intelligent question-answering system needs to combine voice input, image information and medical text database to provide answers. The multimodal intelligent question-answering system needs to process data in multiple modes (such as voice, image, text), switch between multiple tasks (such as speech recognition, image classification, text generation, etc.), and ensure that the memory of old task knowledge is not forgotten. In this scenario, the tasks of the full-modal large model mainly include the following categories.

[0059] Speech recognition task: Convert the user’s speech input into text. The full-modal large model needs to extract speech features from the speech modality and generate corresponding text content.

[0060] Image classification task: classify images uploaded by users and identify the content in the images.

[0061] Text generation task: Generate natural language answers based on input text questions and image descriptions.

[0062] Based on the same inventive concept, the embodiment of the present application also provides a training device for implementing the training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more training device embodiments provided below can refer to the limitations on the training method above, and will not be repeated here.

[0063] In an exemplary embodiment, Figure 5 As shown, a training device for a full-modal large model is provided, including: an old data storage module 501, a joint training module 502, a related data extraction module 503 and a model optimization module 504.

[0064] The old data storage module 501 is used to store sample data of old tasks using the experience replay pool, wherein the old tasks are tasks that have been learned by the full-modality large model, and the sample data include data of multiple modalities.

[0065] The joint training module 502 is used to perform multiple rounds of training on the omnimodal large model when the omnimodal large model learns a new task based on the sample data of the new task and the sample data of the old task stored in the experience replay pool, and during the training process, dynamically adjust the ratio of the sample data of the new task to the sample data of the old task based on the accuracy of the omnimodal large model on the sample data of the old task to obtain the omnimodal large model after joint training.

[0066] The relevant data extraction module 503 is used to select sample data of old tasks whose relevance to the new task is greater than a set threshold from the experience replay pool as old task samples.

[0067] The model optimization module 504 is used to perform multiple rounds of training on the jointly trained full-modal large model according to the old task samples and the sample data of the new task to obtain a trained full-modal large model.

[0068] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data of the experience replay pool and new tasks. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a full-modal large model is implemented.

[0069] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0070] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0071] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0073] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0075] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0076] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A training method for a large omnimodal model, characterized in that: The training method of the full-modal large model includes: The experience replay pool is used to store sample data of old tasks, specifically including: using the experience replay pool to store data of each modality in layers, and storing the data in groups in each layer according to the relevance of the old tasks; wherein the old tasks are tasks that have been learned by the full-modality large model, and the sample data include data of multiple modalities; When the omnimodal large model learns a new task, the omnimodal large model is trained for multiple rounds according to the sample data of the new task and the sample data of the old task stored in the experience replay pool, and during the training process, the ratio of the sample data of the new task to the sample data of the old task is dynamically adjusted according to the accuracy of the omnimodal large model on the sample data of the old task, so as to obtain the omnimodal large model after joint training; Select sample data of old tasks whose correlation with the new task is greater than a set threshold from the experience replay pool as old task samples; perform multiple rounds of training on the jointly trained full-modal large model according to the old task samples and the sample data of the new task to obtain a trained full-modal large model; specifically, randomly extract α×B sample data of old tasks from the experience replay pool, and at the same time, by calculating the characteristic KL divergence of the sample data of the new task and the sample data of the old task, give priority to the sample data with high modality correlation with the new task; randomly extract (1-α)×B samples from the sample data of the new task of each modality, mix the extracted old task samples and the new task samples to form a complete training batch, and use these samples to train the full-modal large model; wherein B is the sample batch size, and α is the proportion of the sample data of the old task; The data of multiple modalities are speech, image and text; the new task and the old task both include speech recognition task, image classification task and text generation task; the full-modal large model is a multi-modal intelligent question-answering system; The multimodal intelligent question-answering system is used to receive voice, images and text, and perform voice recognition tasks based on the voice to convert the voice into text, perform image classification tasks based on the images to classify the images, perform text generation tasks based on the text and image classification results to generate natural language answers.

2. The training method of the full-modal large model according to claim 1 is characterized in that: Use the following formula to determine the t The proportion of sample data of old tasks during round training: ； in, For the t The proportion of sample data of old tasks during round training, is the proportion of sample data of the old task at the beginning, For the t The accuracy of the full-modal large model on the sample data of the old task during the round of training, is the standard accuracy of the full-modal large model on sample data of the old task before learning the new task; No. t The proportion of sample data of new tasks in the round of training is .

3. The training method of the full-modal large model according to claim 1 is characterized in that: The loss function when performing multiple rounds of training on the full-modal large model is: ； in, is the joint loss value, For the i The old task loss of this modality, For the i Sample data of old tasks of different modalities, For the i The new task loss of the modality, For the i Sample data of new tasks of different modalities, is the proportion of sample data of the old task, I is the modal number.

4. The training method of the full-modal large model according to claim 1 is characterized in that: The loss function for multiple rounds of training of the joint-trained full-modal large model is: ； in, is the dependent loss value, For the i The loss of old task samples of this modality, For the i old task samples of different modalities, For the i The new task loss of the modality, For the i Sample data of new tasks of different modalities, is the proportion of sample data of the old task, I is the modal number.

5. A training device for a large omnimodal model, applied to the training method for a large omnimodal model according to any one of claims 1 to 4, characterized in that: The training device of the full-modal large model comprises: An old data storage module is used to store sample data of old tasks using an experience replay pool; wherein the old tasks are tasks that have been learned by the full-modal large model, and the sample data includes data of multiple modalities; The joint training module is used to perform multiple rounds of training on the omnimodal large model when the omnimodal large model learns a new task based on the sample data of the new task and the sample data of the old task stored in the experience replay pool, and dynamically adjust the ratio of the sample data of the new task to the sample data of the old task according to the accuracy of the omnimodal large model on the sample data of the old task during the training process to obtain the omnimodal large model after joint training.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method for the full-modal large model described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the full-modal large model described in any one of claims 1-4.

Images