Multimodal model training method and device, data processing method and device

Abstract

This specification provides a multimodal model training method and apparatus, and a data processing method and apparatus. The multimodal model training method includes: determining a multimodal model to be trained and a corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained in different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained in different modalities; determining a model parallel training strategy corresponding to each model to be trained in different modalities based on multiple model training devices, the models to be trained in different modalities, and / or the training dataset; and performing parallel training on each model to be trained in different modalities using the multiple model training devices based on the training dataset and the model parallel training strategies corresponding to each model to be trained in different modalities to obtain a trained multimodal model.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a multimodal model training method. Background Technology

[0002] With the continuous development of artificial intelligence technology, neural network models can be applied to various types of scenarios to perform various types of data processing tasks; for example, using trained multimodal models to process multimodal data.

[0003] Currently, in the process of training multimodal models, since multimodal models include multiple sub-models with different modalities, the conventional model training methods result in low training efficiency. Therefore, how to improve the training efficiency of multimodal models has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a multimodal model training method. One or more embodiments of this specification also relate to a data processing method, a multimodal model training apparatus, a data processing device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a multimodal model training method is provided, comprising:

[0006] Determine the multimodal model to be trained and the corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained with different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained with different modalities;

[0007] Based on multiple model training devices, the models to be trained in each modality, and / or the training dataset, determine the model parallel training strategy corresponding to the models to be trained in each modality, wherein the model parallel training strategies corresponding to the models to be trained in each modality are different.

[0008] Based on the training dataset and the parallel training strategy corresponding to the model to be trained for each modality, the multiple model training devices are used to train the model to be trained for each modality in parallel to obtain the trained multimodal model.

[0009] According to a second aspect of the embodiments of this specification, a multimodal model training apparatus is provided, comprising:

[0010] The training data determination module is configured to determine the multimodal model to be trained and the corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained with different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained with different modalities.

[0011] The training strategy determination module is configured to determine the parallel training strategy corresponding to the model to be trained in each modality based on multiple model training devices, the model to be trained in each modality, and / or the training dataset, wherein the parallel training strategies corresponding to the model to be trained in each modality are different.

[0012] The model training module is configured to perform parallel training on the models to be trained in each modality using the multiple model training devices, based on the training dataset and the parallel training strategy corresponding to the models to be trained in each modality, to obtain the trained multimodal model.

[0013] According to a third aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0014] A multimodal task processing request is determined, wherein the multimodal task processing request carries visual data or language data;

[0015] The visual data or the language data is input into the multimodal model to obtain the language data corresponding to the visual data, or the visual data corresponding to the language data.

[0016] The multimodal model is obtained by training using the multimodal model training method described above.

[0017] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0018] The request determination module is configured to determine a multimodal task processing request, wherein the multimodal task processing request carries visual data or language data.

[0019] The data acquisition module is configured to input the visual data or the language data into the multimodal model to obtain language data corresponding to the visual data, or visual data corresponding to the language data.

[0020] The multimodal model is obtained by training using the multimodal model training method described above.

[0021] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0022] Memory and processor;

[0023] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-described multimodal model training method or data processing method.

[0024] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the multimodal model training method or data processing method described above.

[0025] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal model training method or data processing method described above.

[0026] This specification provides one or more embodiments of a multimodal model training method, comprising: determining a multimodal model to be trained and a corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained in different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained in different modalities; determining a model parallel training strategy corresponding to each model to be trained in different modalities based on multiple model training devices, the models to be trained in different modalities, and / or the training dataset; and performing parallel training on the models to be trained in different modalities using the multiple model training devices based on the training dataset and the model parallel training strategies corresponding to each model to be trained in different modalities to obtain a trained multimodal model.

[0027] Specifically, this multimodal model training method, during the model training process for a multimodal model, firstly, determines the multimodal model to be trained and its corresponding training dataset; secondly, considering that the multimodal model contains multiple training models of different modalities, in order to improve the model training efficiency of each training model, a corresponding parallel training strategy is determined for each training model based on multiple model training devices, the training models of each modality, and / or the training dataset. Moreover, the parallel training strategies corresponding to each training model of different modalities are different, thereby achieving targeted determination of a parallel training strategy that is more in line with the training situation of multimodal models for each training model.

[0028] Finally, based on the training dataset and the parallel training strategies corresponding to the models to be trained in each modality, multiple model training devices are used to train the models to be trained in each modality in parallel, thereby improving the training efficiency of multimodal models and avoiding the low model training efficiency caused by conventional model training methods when training multimodal models. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a multimodal model provided in one embodiment of this specification;

[0030] Figure 2 This is a schematic diagram illustrating the application of a data processing method provided in one embodiment of this specification;

[0031] Figure 3 This is a flowchart illustrating a multimodal model training method provided in one embodiment of this specification;

[0032] Figure 4 This is a flowchart illustrating the processing procedure of a multimodal model training method provided in one embodiment of this specification.

[0033] Figure 5 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0034] Figure 6 This is a schematic diagram of the structure of a multimodal model training device provided in one embodiment of this specification;

[0035] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0036] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0037] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0038] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0040] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0041] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0042] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0043] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0044] Data Parallelism (DP): Data parallelism is a parallel strategy for model training. In this strategy, training data is divided into multiple batches, and each batch is sent to a different computing device (e.g., GPU). Each device has a copy of the model and independently computes gradients on its own batch of data. Afterward, the gradients from all devices are aggregated (usually through an All-Reduce operation), and the model parameters are updated.

[0045] Tensor parallelism (TP) is a model parallelism technique that partitions the model's weight matrix or other tensors into smaller parts and distributes these parts across different computing devices for processing. This effectively reduces the memory required per device. Common tensor parallelism methods include row parallelism and column parallelism.

[0046] Pipeline Parallelism (PP): Pipeline parallelism is a parallel strategy for model training in which the model is divided into a series of stages, each of which is placed on one or a group of devices (such as GPUs). Data batches pass through these stages in a pipeline manner, with each stage completing its assigned task and passing the result to the next stage.

[0047] Context Parallelism (CP): Context parallelism is a parallelization technique used in model training, primarily for training large language models (LLMs). Context parallelism can segment all inputs and outputs along the sequence dimension, thereby optimizing the computation of the self-attention module. By parallelizing processing along the sequence dimension, it reduces the demand for GPU memory, thus improving training efficiency.

[0048] 3D Parallelism (3DParallelism): Also known as 3D parallelism, 3D parallelism is a hybrid parallelism approach that combines data parallelism, tensor parallelism, and pipelined parallelism. It leverages the advantages of these three parallelism strategies simultaneously within a three-dimensional parallel space to achieve higher parallelism and better resource utilization. This three-dimensional parallelism approach distributes the computational load across multiple dimensions, thereby making better use of large-scale computing resources.

[0049] Multimodal large model (MMB) is a machine learning technique based on deep learning. Its core idea is to fuse different media data (such as text, images, audio, and video) and achieve more intelligent information processing by learning the relationships between different modalities. In MMB, data from different modalities are preprocessed and then input into a deep neural network. After multiple layers of feature extraction and fusion, the final output is the corresponding result.

[0050] Data context refers to the contextual information that a model relies on when processing input data. Context learning, proposed in Large Scale Language Models (LMMs), refers to the underlying patterns, structures, and semantic knowledge that a model learns during the pre-training phase from large amounts of unlabeled text data. This knowledge enables the model to output the desired results based on provided contextual examples when facing different downstream tasks, without needing to adjust model parameters.

[0051] With the continuous development of artificial intelligence technology, neural network models can be applied to various types of scenarios to perform various data processing tasks; for example, using trained multimodal models to process multimodal data. Currently, in the process of training multimodal models, because multimodal models include multiple sub-models with different modalities, using conventional model training methods leads to low model training efficiency.

[0052] For example, there are significant shortcomings in current multimodal pre-training, especially for scenarios that include both visual and language models. Figure 1This is a schematic diagram of a multimodal model provided in one embodiment of this specification. Figure 1 (a), (b), and (c) in the diagram correspond to three different vision-language multimodal models, each with its own modal interactions. In (a), the model's visual module is larger than its text module; for example, the CLIP model, driven by vision-centric requirements, resulting in a larger visual submodule (i.e., the visual module). In (b), the model's visual module is equal to its text module; for example, the CoCa model, which uses equal-sized visual and text submodules (i.e., the text module). In (c), the model's text module is larger than its visual module; for example, the LiT model, whose text submodule has more parameters. It should be noted that the size of the text and visual modules represents the size of their network layers.

[0053] In response to the above Figure 1 Current multimodal model pre-training solutions do not consider the scenario where more than two models are being trained in a multimodal environment. They only provide a single three-dimensional parallel strategy for training, resulting in low training efficiency. Three-dimensional parallelism (3D parallelism) refers to Tensor parallelism, pipeline parallelism, and data parallelism. The efficiency of pipeline parallelism heavily relies on the balanced computation of each stage within the pipeline. However, in multimodal scenarios, it is difficult to find a balanced way to separate the visual and language models, leading to low pipeline parallelism efficiency. Furthermore, the visual part requires image reading, resulting in an input load an order of magnitude larger than the text input of the language model, making parallel processing even more complex.

[0054] Based on this, this specification provides a multimodal model training method. One or more embodiments of this specification also relate to a data processing method, a multimodal model training device, a data processing device, and a computing device, which will be described in detail in the following embodiments.

[0055] See Figure 2 , Figure 2 This diagram illustrates an application illustration of a data processing method according to an embodiment of this specification, based on... Figure 2 It can be seen that the client 202 can send a text and image question-and-answer task processing request (i.e., a multimodal task processing request) to the server 104. The multimodal task processing request carries images (visual data) and questions (i.e., language data).

[0056] After receiving the image-text question-answering task processing request, the server 204 inputs the image and question into the visual-language multimodal model for processing, obtains the answer corresponding to the image and question, and sends the answer to the client 202; thus realizing efficient and accurate processing of multimodal tasks using the visual-language multimodal model.

[0057] See Figure 3 , Figure 3 A flowchart of a multimodal model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0058] Step 302: Determine the multimodal model to be trained and the corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained with different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained with different modalities.

[0059] The model to be trained can be understood as a sub-model contained in the multimodal model to be trained; or one or more network layers contained in the multimodal model to be trained; the multiple different modal models to be trained can be understood as models to be trained that process different types of modal data, such as a language model to be trained that can process text data, language data, etc., or a visual model to be trained that can process image data, graphics data, etc.

[0060] A sample pair can be constructed from training data of two different modalities; for example, a text pair (sample pair) includes an image data and the corresponding text description information of the image; the modality of the training data contained in the sample pair can be determined according to the data modality type that the multimodal model can process; for example, if the multimodal model is a visual-language multimodal model, the sample pair can contain training data of the visual modality (such as image data) and training data of the language modality (such as text data).

[0061] Step 304: Based on multiple model training devices, the models to be trained in each modality, and / or the training dataset, determine the model parallel training strategy corresponding to the models to be trained in each modality, wherein the model parallel training strategies corresponding to the models to be trained in each modality are different.

[0062] Here, model training equipment can be understood as equipment capable of training multimodal models. For example, the model training equipment can be a CPU, GPU, or a server, client, or other similar devices.

[0063] Model parallel training strategy can be understood as a strategy that uses multiple model training devices to train multimodal models in parallel. This model parallel training strategy includes, but is not limited to, data parallelism, model parallelism, pipeline parallelism, tensor parallelism, 3D parallelism, context parallelism and other training strategies.

[0064] It should be noted that the parallel training strategies for the models to be trained in different modalities can be different or the same.

[0065] In one or more embodiments provided in this specification, in order to improve the training efficiency of multimodal models, a corresponding parallel training strategy for each modality's training model is determined; the specific implementation includes steps one and two:

[0066] Step 1: Determining the parallel training strategy for each modality's training model based on multiple model training devices, the models to be trained in each modality, and / or the training dataset, includes:

[0067] Based on the modality of the model to be trained for each modality, determine the model training strategy set corresponding to the model to be trained for each modality, and determine the training data corresponding to the model to be trained for each modality from the training dataset.

[0068] Among them, the modes of the training models to be trained in each mode are different, the training data corresponding to the training models to be trained in each mode are different, and the initial model training strategies corresponding to the training models to be trained in each mode are also different.

[0069] In this context, "modality" can be understood as different forms or types of data representation. For example, in the fields of artificial intelligence and machine learning, the data forms corresponding to various modalities include, but are not limited to, visual, linguistic, textual, image, audio, and video data. That is to say, a modality can be understood as the data type that a multimodal model can process; for example, a modality could be a visual modality or a linguistic modality.

[0070] A model training strategy set can be understood as a collection of various parallel model training strategies.

[0071] The training data corresponding to the training model for each modality can be understood as the training data used to train the training model for various different modalities; for example, the training data corresponding to the training model for the visual modality can be visual training data (such as image data, video data); the training data corresponding to the training model for the language modality can be language training data (such as audio data, text data).

[0072] The initial model training strategy can be understood as the default or initial model training strategy set for each modality of the model to be trained. For example, the initial model training strategy for the visual model to be trained can be a data parallel training strategy; the initial training strategy for the language model to be trained can be a three-dimensional parallel training strategy.

[0073] In the embodiments of this specification, the modality of the model to be trained in each modality can be used to determine the corresponding training data for the model to be trained in each modality, thereby facilitating subsequent model training for the model to be trained in each modality based on the training data of the corresponding modality.

[0074] In one or more embodiments provided in this specification, the plurality of different modal training models include a visual model to be trained and a language model to be trained;

[0075] The step of determining the initial model training strategy corresponding to the model to be trained for each modality based on the modality of the model to be trained for each modality, and determining the training data corresponding to the model to be trained for each modality from the training dataset, includes:

[0076] Based on the visual modality of the visual model to be trained, determine the visual model training strategy set corresponding to the visual model to be trained, and determine the visual training data corresponding to the visual model to be trained from the training dataset; and

[0077] Based on the language modality of the language model to be trained, determine the language model training strategy set corresponding to the language model to be trained, and determine the language training data corresponding to the language model to be trained from the training dataset.

[0078] The visual model to be trained can be understood as the visual sub-model in the visual-language multimodal model; this visual sub-model can be used to perform visual processing tasks such as image recognition, image feature extraction, and image segmentation; for example, the visual model to be trained can be a convolutional neural network (CNNs) or a vision transformer (ViTs).

[0079] The language model to be trained can be understood as the language sub-model in the vision-language multimodal model; this language sub-model can be used to perform language processing tasks such as text feature extraction and language reasoning; for example, the language model to be trained can be Transformers, LLM, etc.

[0080] Visual modalities can refer to data types acquired through visual perception, including images and videos; linguistic modalities can refer to data types acquired through language perception, including text and audio.

[0081] A visual model training strategy set can be understood as a collection of various parallel training strategies for a visual model to be trained. Similarly, a language model training strategy set can be understood as a collection of various parallel training strategies for a language model to be trained.

[0082] The visual training data for the visual model to be trained can be image data, video data, etc.; the language training data for the language model to be trained can be audio data, text data, etc.

[0083] In the embodiments of this specification, the visual modality of the visual model to be trained can be used to determine the corresponding visual training data for the visual model to be trained, thereby facilitating the subsequent targeted model training of the visual model to be trained based on the visual training data of the visual modality; and the language modality of the language model to be trained can be used to determine the corresponding language training data for the language model to be trained, thereby facilitating the subsequent targeted model training of the language model to be trained based on the language training data of the language modality.

[0084] Step 2: Identify multiple model training devices, and based on the device information of the multiple model training devices, the model information of the models to be trained in each modality, and / or the data information of the training data corresponding to the models to be trained in each modality, determine the parallel training strategy corresponding to the models to be trained in each modality from the model training strategy set corresponding to the models to be trained in each modality.

[0085] The device information includes, but is not limited to, the number of devices for multiple model training devices and the available device resources. The available device resources refer to the information on device resources that can be used for model training, which can be idle computing resources or idle storage resources. Correspondingly, the available device resources information can be the idle computing resources and idle storage resources of multiple model training devices.

[0086] Model information includes, but is not limited to, model parameter information and model depth information of the model to be trained; the model parameter information can be understood as the amount of data of model weight data and the amount of data of model matrix data; the model depth information can be understood as the size of model depth and the number of network layers in the model to be trained.

[0087] Data information includes, but is not limited to, the amount of training data and the data context information of the training data; the data context information can refer to the amount of data in the data context or the size of the data context.

[0088] Specifically, in determining the corresponding parallel training strategy for each modality of the model to be trained, this method first determines multiple model training devices used for model training; second, it determines the device information of the multiple model training devices, the model information of each modality of the model to be trained, and / or the data information of the training data corresponding to each modality of the model to be trained; finally, it determines the model training strategy set corresponding to each modality of the model to be trained, and determines the corresponding parallel training strategy for each modality of the model to be trained from the model training strategy set corresponding to each modality of the model to be trained, based on the device information, model information and / or the data information of each training data.

[0089] Taking the application of the multimodal model training method provided in this manual in the training scenario of a vision-language multimodal model as an example, this multimodal model training method will be explained; wherein, the model training device is a GPU, and the models to be trained for each modality are the vision sub-model and the language sub-model to be trained.

[0090] Based on this, in determining the corresponding parallel training strategies for the visual sub-model and the language sub-model, this method first identifies the multiple GPUs used for model training; second, it determines the number of GPUs (device information), the model depth information (i.e., model information) of the visual sub-model and / or the language sub-model, and the data information of the training data corresponding to the visual sub-model and the language sub-model; finally, based on the number of GPUs, the model depth information, and / or the data information, it determines the corresponding parallel training strategies for the visual sub-model and the language sub-model from the model training strategy sets corresponding to the visual sub-model and the language sub-model, respectively.

[0091] As can be seen from the above embodiments, in the process of determining the corresponding parallel training strategy for each modality of the training model, this method determines the corresponding parallel training strategy for each modality of the training model based on the device information of multiple model training devices, the model information of each modality of the training model, and / or the data information of the training data corresponding to each modality of the training model. This achieves targeted determination of a parallel training strategy that is more in line with the training situation of multimodal models for each modality of the training model, thereby improving the efficiency of subsequent model training.

[0092] In one or more embodiments provided in this specification, the visual model training strategy set includes a data parallel training strategy, a tensor parallel training strategy, a pipeline parallel training strategy, and a context parallel training strategy, and the language model training strategy set includes a data parallel training strategy, a tensor parallel training strategy, and a pipeline parallel training strategy.

[0093] The step of determining the parallel training strategy for each modality's training model from the model training strategy set corresponding to each modality's training model, based on the device information of the multiple model training devices, the model information of each modality's training model, and / or the data information of the training data corresponding to each modality's training model, includes:

[0094] Based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and / or the data information of the visual training data, at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; and

[0095] Based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained.

[0096] Among them, data parallel training strategy can refer to a strategy of training the model using data parallelism; tensor parallel training strategy can refer to a strategy of training the model using tensor parallelism; pipeline parallel training strategy can refer to a strategy of training the model using pipeline parallelism; and context parallel training strategy can refer to a strategy of training the model using context parallelism.

[0097] Parallel training strategy for visual models refers to a model parallel training strategy that performs parallel training on the visual model to be trained; parallel training strategy for language models refers to a model parallel training strategy that performs parallel training on the language model to be trained.

[0098] Following the previous example, in determining the corresponding parallel training strategies for the visual sub-model and the language sub-model, this method can adopt different methods for determining the parallel training strategies for the visual sub-model and the language sub-model.

[0099] Specifically, for the visual sub-model:

[0100] First, the multiple GPUs used for training the visual sub-model can be identified;

[0101] Secondly, determine the number of GPUs (device information), the model depth information of the visual sub-model (i.e., model information), and / or the data information of the visual training data corresponding to the visual sub-model;

[0102] Finally, based on the number of GPUs, model depth information, and / or data information, at least one of the following strategies—data parallel training strategy, tensor parallel training strategy, pipeline parallel training strategy, and context parallel training strategy—is determined as the visual model parallel training strategy corresponding to the visual sub-model.

[0103] Specifically, for the language sub-model:

[0104] First, the multiple GPUs used for training the language sub-model can be identified;

[0105] Secondly, determine the number of GPUs (device information), the model depth information of the language sub-model (i.e., model information), and / or the data information of the language training data corresponding to the language sub-model;

[0106] Finally, based on the number of GPUs, model depth information, and / or data information, at least one of the data parallel training strategy, tensor parallel training strategy, and pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language sub-model.

[0107] As can be seen from the above embodiments, in the process of determining the corresponding parallel training strategies for the visual model and the language model to be trained, this method determines the parallel training strategies for the visual model and the language model to be trained based on the device information of multiple model training devices, the model information of the visual model and the language model to be trained, and / or the data information of the training data corresponding to the visual model and the language model to be trained. This achieves targeted determination of the parallel training strategies for the visual model and the language model to be trained that are more in line with the training situation of multimodal models, thereby improving the efficiency of subsequent multimodal model training.

[0108] In one embodiment provided in this specification, the plurality of different modal training models include a visual model to be trained and a language model to be trained;

[0109] The step of determining the parallel training strategy for each modality's training model based on multiple model training devices, the models to be trained in each modality, and / or the training dataset includes:

[0110] Based on multiple model training devices, the visual model to be trained, and / or the visual training data corresponding to the visual model to be trained in the training dataset, a parallel training strategy for the visual model to be trained is determined; and

[0111] Based on multiple model training devices, the language model to be trained, and / or the language training data corresponding to the language model to be trained in the training dataset, determine the parallel training strategy for the language model to be trained.

[0112] For explanations of this embodiment, please refer to the corresponding explanations in one or more of the above embodiments, which will not be elaborated upon here.

[0113] In one embodiment provided in this specification, determining a parallel training strategy for the visual model corresponding to the visual model to be trained, based on multiple model training devices, the visual model to be trained, and / or the visual training data corresponding to the visual model to be trained in the training dataset, includes:

[0114] Based on the visual modality of the visual model to be trained, determine the visual model training strategy set corresponding to the visual model to be trained, and determine the visual training data corresponding to the visual model to be trained from the training dataset.

[0115] Based on the device information of the multiple model training devices, the model information of the visual model to be trained, and / or the data information of the visual training data, at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0116] Accordingly, determining the parallel training strategy for the language model corresponding to the language model to be trained based on multiple model training devices, the language model to be trained, and / or the language training data corresponding to the language model to be trained in the training dataset includes:

[0117] Based on the language modality of the language model to be trained, determine the language model training strategy set corresponding to the language model to be trained, and determine the language training data corresponding to the language model to be trained from the training dataset;

[0118] Based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained.

[0119] For explanations of this embodiment, please refer to the corresponding explanations in one or more of the above embodiments, which will not be elaborated upon here.

[0120] In one or more embodiments provided in this specification, determining at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy as the visual model parallel training strategy corresponding to the visual model to be trained, based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and the data information of the visual training data, includes:

[0121] If, based on the device information of the multiple model training devices, it is determined that the computing resources of the multiple model training devices are greater than a preset resource threshold, the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy are determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0122] Here, computing resources can be understood as the number of devices used for multiple model training exercises or the percentage of idle computing resources. For example, when the model training device is a GPU, computing resources can be the number of GPUs or the percentage of idle CPU resources. Preset resource thresholds can be set according to the actual application scenario, and will not be overly restricted here.

[0123] Following the previous example, during the parallel training of the visual sub-model, after determining 32 GPUs for the visual model to be trained, if the number of GPUs (i.e., computing resources) is greater than 24 (preset resource threshold), then the data parallel training strategy, tensor parallel training strategy, and pipeline parallel training strategy can be determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0124] In one or more embodiments provided in this specification, determining at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy as the visual model parallel training strategy corresponding to the visual model to be trained, based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and the data information of the visual training data, includes:

[0125] If, based on the model information of the visual model to be trained, it is determined that the model depth of the visual model to be trained is greater than a preset depth threshold, then the pipeline parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0126] The preset depth threshold can be set according to the actual application scenario, and will not be restricted in detail here.

[0127] Using the previous example, during the parallel training of the visual sub-model, if the number of network layers (i.e. model depth) in the visual sub-model is 100 layers, which is greater than 70 layers (i.e., the preset depth threshold), then the visual sub-model is considered large. In the case of a large visual sub-model, a pipeline parallelism (PP) training strategy can be used to train the visual sub-model (i.e., the visual model).

[0128] It should be noted that the method of training the visual model using the pipelined parallel training strategy is as follows: the number of model layers of the visual model is divided according to the number of preset computing devices (such as GPUs); for example, if the visual model has 100 model layers and there are 4 computing devices, then the number of model layers of the visual model is divided into 4 equal parts, and the model is trained on each of the computing devices.

[0129] In one or more embodiments provided in this specification, determining at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy as the visual model parallel training strategy corresponding to the visual model to be trained, based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and the data information of the visual training data, includes:

[0130] If, based on the data information of the visual training data, it is determined that the data context of the visual training data is greater than a preset context threshold, the context-parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or

[0131] The preset context threshold can be set according to the actual application scenario, and will not be restricted in detail here.

[0132] Using the previous example, during the parallel training of a visual model, if it is determined that the context of a certain training data in the training dataset (visual training data) corresponding to the visual model is greater than a preset context threshold (e.g., the preset context threshold can be set to 128kb), then the context parallelism (CP) training strategy can be used to train the visual model.

[0133] It should be noted that the method of training the visual model using the context parallel training strategy is as follows: if the context of the visual model is 256kb and there are 4 computing devices, then the context of the visual model is divided into 4 parts, and 256 / 4 = 64kb of model training is performed on each computing device.

[0134] In one or more embodiments provided in this specification, determining at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy as the visual model parallel training strategy corresponding to the visual model to be trained, based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and the data information of the visual training data, includes:

[0135] If, based on the model information of the visual model to be trained, it is determined that the model parameters of the model to be trained are greater than a preset parameter threshold, then the tensor parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or

[0136] The model parameters can be the amount of data for model weight parameters, the amount of data for model matrix parameters, etc.; the preset parameter thresholds can be set according to the actual application scenario, and there are no further restrictions here.

[0137] Following the previous example, during the parallel training of the visual model, if it is determined that the amount of data (7b) of the model parameters of the visual model is greater than the preset parameter threshold (e.g., the preset parameter threshold can be 5b), then the tensor parallel training strategy can be used to train the visual model.

[0138] It should be noted that the tensor parallel training strategy is used to train the visual model as follows: if the visual model has 7 bytes of parameters and there are 4 computing devices, then the model parameters of the visual model are divided into 4 parts, and the model is trained on each of the computing devices.

[0139] In one or more embodiments provided in this specification, determining at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy as the visual model parallel training strategy corresponding to the visual model to be trained, based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and the data information of the visual training data, includes:

[0140] If, based on the data information of the visual training data, the amount of visual training data is determined to be greater than a preset data amount threshold, the data parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0141] The preset data volume threshold can be set according to the actual application scenario, and will not be restricted in detail here.

[0142] Using the previous example, if the amount of training data for the visual model is greater than a preset data threshold, then a data-parallel training strategy can be used to train the visual model.

[0143] It should be noted that the method of training the visual model using the data parallel training strategy is as follows: if the amount of training data for the visual model is large and there are 4 computing devices, then the training data is divided into 4 parts, and the model is trained on each of the computing devices respectively.

[0144] As can be seen from the above embodiments, the multimodal model training method provided in this specification, during the model training process for the visual model to be trained, can determine different parallel training strategies for the visual model to be trained based on multiple model training devices, the visual model to be trained, and / or the visual training dataset. This achieves the ability to determine a more suitable parallel training strategy for the visual model to be trained, tailored to the specific circumstances of multimodal model training. Subsequently, the visual model to be trained can be efficiently trained in parallel according to the parallel training strategy, thereby improving the training efficiency of the multimodal model.

[0145] In one or more embodiments provided in this specification, determining at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy as the language model parallel training strategy corresponding to the language model to be trained, based on the device information of the plurality of model training devices, the model information of the language model to be trained, and the data information of the language training data, includes:

[0146] If, based on the device information of the multiple model training devices, it is determined that the computing resources of the multiple model training devices are greater than a preset resource threshold, the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy are determined as the language model parallel training strategy corresponding to the language model to be trained.

[0147] Following the previous example, during the parallel training of the language sub-model (i.e., the language model to be trained), after determining 32 GPUs for the language model (i.e., the language sub-model), if the number of GPUs (i.e., computing resources) is greater than 24 (a preset resource threshold), then the data parallel training strategy, tensor parallel training strategy, and pipeline parallel training strategy can be determined as the language model parallel training strategy corresponding to the language model to be trained. That is, the three-dimensional parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained; this three-dimensional parallel training strategy can be a combination of the data parallel training strategy, tensor parallel training strategy, and pipeline parallel training strategy.

[0148] As can be seen from the above embodiments, the multimodal model training method provided in this specification, during the model training process for the language model to be trained, can determine different parallel training strategies for the language model to be trained based on multiple model training devices, the language model to be trained, and / or the language training dataset. This achieves the ability to determine a more suitable parallel training strategy for the language model to be trained based on the actual situation, thus improving the training efficiency of the multimodal model. Subsequently, the language model to be trained can be efficiently trained in parallel according to the parallel training strategy, thereby improving the training efficiency of the multimodal model.

[0149] Step 306: Based on the training dataset and the parallel training strategy corresponding to the training models of each modality, use the multiple model training devices to train the training models of each modality in parallel to obtain the trained multimodal models.

[0150] In one or more embodiments provided in this specification, the step of using the multiple model training devices to perform parallel training on the models to be trained in each modality, based on the training dataset and the model parallel training strategy corresponding to the models to be trained in each modality, to obtain the trained multimodal model, includes:

[0151] Based on the visual training data and the visual model parallel training strategy, the visual model to be trained is trained in parallel using the multiple model training devices to obtain the visual training data features of the visual training data.

[0152] Based on the language training data, the visual training data features, and the language model parallel training strategy, the language model to be trained is trained in parallel using the multiple model training devices to obtain the language training data features of the language training data.

[0153] Based on the visual training data features and the language training data features, the visual model to be trained and the language model to be trained are trained to obtain the trained multimodal model.

[0154] Following the previous example, in the process of training a multimodal model, firstly, the visual model can be trained based on the visual training data and the parallel training strategy of the visual model. During the process of reading the training data, the visual model can segment the training data. After the visual model completes the processing of the training data, it will merge the data output by the visual model with the language training data, and then hand the merged data over to the downstream (i.e., the language model) for processing.

[0155] Assuming there are 8 GPU devices, the visual model training part is divided as follows: dp=8 (the data parallel training strategy divides the training data into 8 parts). The data output by each GPU is then merged and handed over to the downstream language model for processing.

[0156] Secondly, based on the parallel training strategy of the language model, the features of the language training data and the visual training data are input into the language model to obtain the output data of the language model (i.e., the features of the language training data).

[0157] Specifically, the language model is trained using data-parallel training, tensor-parallel training, and pipeline-parallel training strategies as follows:

[0158] A data-parallel training strategy is adopted to segment the input data of the language model. The segmentation method is: dp=2 (the data-parallel training strategy divides the training data into 2 parts); at this time, the visual model part merges the data of every 4 GPUs, and then the merged data is given to a certain dp in the language model.

[0159] Tensor parallel training and pipeline parallel training strategies are employed to partition the tensors and network layers of the language model, with partitioning methods of tp=4 and pp=4. Here, tp=4 indicates a tensor parallelism of 4, and pp=4 indicates a pipeline parallelism of 4.

[0160] Finally, based on the data output by the visual and language models, the parameters of the visual and language models are adjusted to obtain the trained visual-language multimodal model.

[0161] In the above embodiments, based on the training dataset and the parallel training strategy corresponding to the training models of each modality, multiple model training devices are used to train the training models of each modality in parallel, thereby improving the training efficiency of multimodal models and avoiding the low model training efficiency caused by conventional model training methods when training multimodal models.

[0162] In one or more embodiments provided in this specification, the step of using the multiple model training devices to perform parallel training on the models to be trained in each modality, based on the training dataset and the model parallel training strategy corresponding to the models to be trained in each modality, to obtain the trained multimodal model, includes:

[0163] Based on the language training data and the language model parallel training strategy, the language model to be trained is trained in parallel using the multiple model training devices to obtain the language training data features of the language training data.

[0164] Based on the visual training data, the language training data features, and the visual model parallel training strategy, the visual model to be trained is trained in parallel using the multiple model training devices to obtain the visual training data features of the visual training data.

[0165] Based on the visual training data features and the language training data features, the visual model to be trained and the language model to be trained are trained to obtain the trained multimodal model.

[0166] Following the previous example, firstly, according to the parallel training strategy of the language model, the language training data and visual training data features are input into the language model processing to obtain the language model output data (i.e., language training data features).

[0167] Specifically, the language model is trained using data-parallel training, tensor-parallel training, and pipeline-parallel training strategies as follows:

[0168] A data-parallel training strategy is adopted to segment the training data of the input language model. The segmentation method is: dp=2 (the data-parallel training strategy divides the training data into 8 parts);

[0169] Tensor parallel training and pipeline parallel training strategies are employed to partition the tensors and network layers of the language model, with partitioning methods of tp=4 and pp=4. Here, tp=4 indicates a tensor parallelism of 4, and pp=4 indicates a pipeline parallelism of 4.

[0170] Secondly, the visual model is trained using a parallel training strategy for the visual model, the output data of the language model, and the visual training data to obtain the output data of the visual model.

[0171] Finally, based on the data output by the visual and language models, the parameters of the visual and language models are adjusted to obtain the trained visual-language multimodal model.

[0172] This specification provides one or more embodiments of a multimodal model training method. In the process of training a multimodal model, firstly, the multimodal model to be trained and its corresponding training dataset are determined. Secondly, considering that the multimodal model contains multiple training models of different modalities, in order to improve the training efficiency of each training model, a corresponding parallel training strategy is determined for each training model based on multiple training devices, the training models of each modality, and / or the training dataset. Furthermore, the parallel training strategies for each training model are different, thereby achieving targeted determination of a parallel training strategy more suitable for the multimodal model training situation for each training model.

[0173] Finally, based on the training dataset and the parallel training strategies corresponding to the models to be trained in each modality, multiple model training devices are used to train the models to be trained in each modality in parallel, thereby improving the training efficiency of multimodal models and avoiding the low model training efficiency caused by conventional model training methods when training multimodal models.

[0174] The following is in conjunction with the appendix Figure 4 Taking the application of the multimodal model training method provided in this specification in a visual-language multimodal model training scenario as an example, the multimodal model training method will be further explained. Among other things, Figure 4 A flowchart illustrating the processing procedure of a multimodal model training method provided in one embodiment of this specification is shown.

[0175] based on Figure 4 As can be seen, the multimodal model provided in this specification includes: a vision model and a language model. Language model stage 0, language model stage 2, and language model stage 4 are obtained by segmenting the network layers of the language model, resulting in multiple language model stages, each allocated to a different GPU. Vision model 0, vision model 2, and vision model 4 are copies of the vision model allocated to different GPUs; the model parameters of the vision models on multiple GPUs are identical.

[0176] To address the issues of the visual model requiring data loading (I / O intensive) before computation, and the high weighting and computational complexity of the LLM (language model), this method employs a data parallel strategy for the visual model and a conventional three-dimensional parallel strategy for the language model. By using different distributed strategies based on the characteristics of different models, training becomes more efficient; the specific steps include:

[0177] Step 402: Image input.

[0178] During the training of the visual-language multimodal model, the visual model can be trained based on visual training data and a parallel training strategy for the visual model.

[0179] Specifically, the visual model can segment the training data during the reading process; if the amount of training data for the visual model is large, there are 3 computing devices; the segmentation method for the visual model training part is: dp=3 (the data parallel training strategy divides the training data into 3 parts).

[0180] Step 404: Data-parallel training.

[0181] Specifically, after the data is segmented, the segmented training data is input into the visual models on three GPUs, which then process the data to obtain the output data (feature vectors) of the visual models.

[0182] It should be noted that during the parallel training of visual models, pipeline parallelism (PP), context parallelism (CP), and tensor parallelism (TP) training strategies can also be used to train visual models in parallel.

[0183] Specifically, regarding the pipelined parallel training strategy:

[0184] If the number of network layers (i.e. model depth) in the visual sub-model is 90 layers, which is greater than 70 layers (i.e., the preset depth threshold), then the visual sub-model is considered large. In the case of a large visual sub-model, a pipelined parallel training strategy can be used to train the visual model.

[0185] The method of training a visual model using a pipelined parallel training strategy is as follows: the number of model layers of the visual model is divided according to the number of preset computing devices (such as GPUs); for example, if the visual model has 90 model layers and there are 3 computing devices, the number of model layers of the visual model is divided into 3 equal parts, and the model is trained on each computing device respectively.

[0186] Among them, the context-parallel training strategy is as follows:

[0187] If it is determined that the context of a certain training data in the training dataset corresponding to the visual model is greater than a preset context threshold (e.g., the preset context threshold can be set to 128kb), then the context-parallel training strategy can be used to train the visual model.

[0188] The method of training the visual model using the context parallel training strategy is as follows: if the context of the visual model is 180kb and there are 3 computing devices, then the context of the visual model is divided into 3 parts, and 180 / 3 = 60kb model training is performed on each computing device.

[0189] Specifically, regarding the parallel training strategy for Tensors:

[0190] If the amount of data (7b) of the model parameters of the visual model is greater than the preset parameter threshold (e.g., the preset parameter threshold can be 5b), then the tensor parallel training strategy can be used to train the visual model.

[0191] The method of training a visual model using a tensor parallel training strategy is as follows: if the visual model has 7 bytes of parameters and there are 3 computing devices, then the model parameters of the visual model are divided into 3 parts, and the model is trained on each of the computing devices respectively.

[0192] Based on the aforementioned steps for parallel training of the visual model using data-parallel training strategies, pipelined parallel training strategies, context-parallel training strategies, and tensor parallel training strategies, it can be seen that for the visual model part, this method allows... Figure 4 The visual model (ViT Model) in the program is simply data parallel, which allows all GPUs to participate in reading. Alternatively, pipelined parallel training strategies, context parallel training strategies, and tensor parallel training strategies can be used to train the visual model in parallel. After the visual model is processed, the three-dimensional parallel training phase of the language model is carried out.

[0193] In practice, this method provides three combination methods for different scenarios:

[0194] 1. Visual TP as DP

[0195] 2. Visual pp as dp

[0196] 3. Visual cp as dp

[0197] Using the three methods described above, users can choose tp (tensor parallelism), pp (pipeline parallelism), and cp (context parallelism) in the visual model as training strategies for dp (data parallelism).

[0198] Step 406: Text input.

[0199] Identify the text training data for the language model.

[0200] Step 408: Data merging.

[0201] Specifically, after the visual model completes the processing of the training data, it will collect the data output by the visual model and then merge the collected visual model output data with the text training data of the language model.

[0202] Then, the merged data is handed over to the downstream (i.e., the language model) for processing.

[0203] Step 410: 3D parallel training.

[0204] Specifically, a three-dimensional parallel training strategy, consisting of data-parallel training, tensor-parallel training, and pipeline-parallel training, is employed to train the language model. The specific method is as follows:

[0205] A data-parallel training strategy is adopted to segment the data input to the language model. The segmentation method can be: dp=3 (the data-parallel training strategy divides the training data into 3 parts).

[0206] Tensor parallel training and pipeline parallel training strategies are employed to segment the tensors and network layers of the language model. The segmentation method can be: tp=3, pp=3. Here, tp=3 refers to a tensor parallelism of 3; pp=3 refers to a pipeline parallelism of 3.

[0207] Subsequently, based on the data output by the visual model and the language model, the parameters of the visual model and the language model can be adjusted to obtain the trained visual-language multimodal model.

[0208] Based on the above steps, the multimodal model training method in the embodiments of this specification provides an efficient parallel strategy for multimodal models, used for training multimodal models. This method proposes a hybrid parallel strategy in multimodal model training scenarios to improve training efficiency. Furthermore, in response to the problem of repeated data reading within the visual model DP after adopting hybrid parallelism in multimodal training scenarios, it achieves more efficient model training.

[0209] See Figure 5 , Figure 5 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0210] Step 502: Determine a multimodal task processing request, wherein the multimodal task processing request carries visual data or language data.

[0211] Step 504: Input the visual data or the language data into the multimodal model to obtain the language data corresponding to the visual data or the visual data corresponding to the language data, wherein the multimodal model is trained by the multimodal model training method described above.

[0212] This specification provides one or more embodiments of a data processing method. In the process of processing visual or language data using a multimodal model, it is necessary to train the multimodal model. During the training process, firstly, the multimodal model to be trained and its corresponding training dataset are determined. Secondly, considering that the multimodal model contains multiple models to be trained with different modalities, in order to improve the training efficiency of each model, a corresponding parallel training strategy is determined for each model to be trained based on multiple model training devices, the models to be trained with each modality, and / or the training dataset. Furthermore, the parallel training strategies for each model to be trained are different, thereby achieving targeted determination of a parallel training strategy more suitable for the multimodal model training situation. Finally, based on the training dataset and the parallel training strategies corresponding to each model to be trained, multiple model training devices are used to perform efficient parallel training on each model to be trained, thereby obtaining a multimodal model capable of processing visual or language data.

[0213] Based on this, once the multimodal model is trained, it can be applied to multimodal data processing scenarios to perform multimodal tasks. When a multimodal task processing request carrying visual or linguistic data is received, the visual or linguistic data can be input into the multimodal model to quickly obtain the linguistic data corresponding to the visual data or the visual data corresponding to the linguistic data.

[0214] The above is an illustrative scheme of a data processing method according to this embodiment. It should be noted that the technical solution of this data processing method belongs to the same concept as the technical solution of the multimodal model training method described above. For details not described in detail in the technical solution of the data processing method, please refer to the description of the technical solution of the multimodal model training method described above.

[0215] Corresponding to the above method embodiments, this specification also provides embodiments of a multimodal model training device. Figure 6 A schematic diagram of a multimodal model training device according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0216] The training data determination module 602 is configured to determine the multimodal model to be trained and the corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained with different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained with different modalities.

[0217] The training strategy determination module 604 is configured to determine the parallel training strategy corresponding to the model to be trained for each modality based on multiple model training devices, the model to be trained for each modality, and / or the training dataset, wherein the parallel training strategies corresponding to the model to be trained for each modality are different.

[0218] The model training module 606 is configured to perform parallel training on the models to be trained in each modality using the multiple model training devices according to the training dataset and the model parallel training strategy corresponding to the models to be trained in each modality, so as to obtain the trained multimodal model.

[0219] Optionally, the training strategy determination module 604 is further configured to:

[0220] Based on the modality of the model to be trained for each modality, determine the model training strategy set corresponding to the model to be trained for each modality, and determine the training data corresponding to the model to be trained for each modality from the training dataset.

[0221] Among them, the modes of the training models to be trained in each mode are different, the training data corresponding to the training models to be trained in each mode are different, and the initial model training strategies corresponding to the training models to be trained in each mode are also different.

[0222] Multiple model training devices are identified, and based on the device information of the multiple model training devices, the model information of the model to be trained in each modality, and / or the data information of the training data corresponding to the model to be trained in each modality, a parallel training strategy for the model to be trained in each modality is determined from the model training strategy set corresponding to the model to be trained in each modality.

[0223] Optionally, the multiple different modalities of the models to be trained include a visual model to be trained and a language model to be trained;

[0224] The training strategy determination module 604 is further configured to:

[0225] Based on the visual modality of the visual model to be trained, determine the visual model training strategy set corresponding to the visual model to be trained, and determine the visual training data corresponding to the visual model to be trained from the training dataset; and

[0226] Based on the language modality of the language model to be trained, determine the language model training strategy set corresponding to the language model to be trained, and determine the language training data corresponding to the language model to be trained from the training dataset.

[0227] Optionally, the visual model training strategy set includes data parallel training strategy, tensor parallel training strategy, pipeline parallel training strategy, and context parallel training strategy, and the language model training strategy set includes data parallel training strategy, tensor parallel training strategy, and pipeline parallel training strategy.

[0228] The training strategy determination module 604 is further configured to:

[0229] Based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and / or the data information of the visual training data, at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; and

[0230] Based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained.

[0231] Optionally, the multiple different modalities of the models to be trained include a visual model to be trained and a language model to be trained;

[0232] The training strategy determination module 604 is further configured to:

[0233] Based on multiple model training devices, the visual model to be trained, and / or the visual training data corresponding to the visual model to be trained in the training dataset, a parallel training strategy for the visual model to be trained is determined; and

[0234] Based on multiple model training devices, the language model to be trained, and / or the language training data corresponding to the language model to be trained in the training dataset, determine the parallel training strategy for the language model to be trained.

[0235] Optionally, the training strategy determination module 604 is further configured to:

[0236] Based on the visual modality of the visual model to be trained, determine the visual model training strategy set corresponding to the visual model to be trained, and determine the visual training data corresponding to the visual model to be trained from the training dataset.

[0237] Based on the device information of the multiple model training devices, the model information of the visual model to be trained, and / or the data information of the visual training data, at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0238] Accordingly, the training strategy determination module 604 is further configured to:

[0239] Based on the language modality of the language model to be trained, determine the language model training strategy set corresponding to the language model to be trained, and determine the language training data corresponding to the language model to be trained from the training dataset;

[0240] Based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained.

[0241] Optionally, the training strategy determination module 604 is further configured to:

[0242] If, based on the device information of the multiple model training devices, it is determined that the computing resources of the multiple model training devices exceed a preset resource threshold, then the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy are determined as the visual model parallel training strategy corresponding to the visual model to be trained; or

[0243] If, based on the model information of the visual model to be trained, the model depth of the model to be trained is determined to be greater than a preset depth threshold, then the pipelined parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or

[0244] If, based on the data information of the visual training data, it is determined that the data context of the visual training data is greater than a preset context threshold, the context-parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or

[0245] If, based on the model information of the visual model to be trained, it is determined that the model parameters of the model to be trained are greater than a preset parameter threshold, then the tensor parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or

[0246] If, based on the data information of the visual training data, the amount of visual training data is determined to be greater than a preset data amount threshold, the data parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained.

[0247] Optionally, the training strategy determination module 604 is further configured to:

[0248] If, based on the device information of the multiple model training devices, it is determined that the computing resources of the multiple model training devices are greater than a preset resource threshold, the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy are determined as the language model parallel training strategy corresponding to the language model to be trained.

[0249] Optionally, the model training module 606 is further configured to:

[0250] Based on the visual training data and the visual model parallel training strategy, the visual model to be trained is trained in parallel using the multiple model training devices to obtain the visual training data features of the visual training data.

[0251] Based on the language training data, the visual training data features, and the language model parallel training strategy, the language model to be trained is trained in parallel using the multiple model training devices to obtain the language training data features of the language training data.

[0252] Based on the visual training data features and the language training data features, the visual model to be trained and the language model to be trained are trained to obtain the trained multimodal model.

[0253] Optionally, the model training module 606 is further configured to:

[0254] Based on the language training data and the language model parallel training strategy, the language model to be trained is trained in parallel using the multiple model training devices to obtain the language training data features of the language training data.

[0255] Based on the visual training data, the language training data features, and the visual model parallel training strategy, the visual model to be trained is trained in parallel using the multiple model training devices to obtain the visual training data features of the visual training data.

[0256] Based on the visual training data features and the language training data features, the visual model to be trained and the language model to be trained are trained to obtain the trained multimodal model.

[0257] This specification provides one or more embodiments of a multimodal model training apparatus. During the training of a multimodal model, firstly, the multimodal model to be trained and its corresponding training dataset are determined. Secondly, considering that the multimodal model contains multiple training models of different modalities, in order to improve the training efficiency of each training model, a corresponding parallel training strategy is determined for each training model based on multiple model training devices, the training models of each modality, and / or the training dataset. Furthermore, the parallel training strategies for each training model are different, thereby achieving targeted determination of a parallel training strategy more suitable for the multimodal model training situation for each training model.

[0258] Finally, based on the training dataset and the parallel training strategies corresponding to the models to be trained in each modality, multiple model training devices are used to train the models to be trained in each modality in parallel, thereby improving the training efficiency of multimodal models and avoiding the low model training efficiency caused by conventional model training methods when training multimodal models.

[0259] The above is a schematic scheme of a multimodal model training device according to this embodiment. It should be noted that the technical solution of this multimodal model training device and the technical solution of the multimodal model training method described above belong to the same concept. For details not described in detail in the technical solution of the multimodal model training device, please refer to the description of the technical solution of the multimodal model training method described above.

[0260] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 7 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0261] The request determination module 702 is configured to determine a multimodal task processing request, wherein the multimodal task processing request carries visual data or language data.

[0262] The data acquisition module 704 is configured to input the visual data or the language data into the multimodal model to obtain the language data corresponding to the visual data or the visual data corresponding to the language data, wherein the multimodal model is trained by the multimodal model training method described above.

[0263] This specification provides one or more embodiments of a data processing apparatus. In the process of processing visual or language data using a multimodal model, it is necessary to train the multimodal model. During the training process, firstly, the multimodal model to be trained and its corresponding training dataset are determined. Secondly, considering that the multimodal model contains multiple models to be trained with different modalities, in order to improve the training efficiency of each model, a corresponding parallel training strategy is determined for each model to be trained based on multiple model training devices, the models to be trained with each modality, and / or the training dataset. Furthermore, the parallel training strategies for each model to be trained are different, thereby achieving targeted determination of a parallel training strategy more suitable for the multimodal model training situation. Finally, based on the training dataset and the parallel training strategies corresponding to each model to be trained, multiple model training devices are used to perform efficient parallel training on each model to be trained, thereby obtaining a multimodal model capable of processing visual or language data.

[0264] Based on this, once the multimodal model is trained, it can be applied to multimodal data processing scenarios to perform multimodal tasks. When a multimodal task processing request carrying visual or linguistic data is received, the visual or linguistic data can be input into the multimodal model to quickly obtain the linguistic data corresponding to the visual data or the visual data corresponding to the linguistic data.

[0265] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0266] Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0267] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0268] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0269] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0270] The processor 820 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned multimodal model training method or data processing method.

[0271] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the multimodal model training method or data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the multimodal model training method or data processing method embodiments.

[0272] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described multimodal model training method or data processing method.

[0273] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are described simply because they are substantially similar to the multimodal model training method or data processing method embodiments; relevant parts can be referred to in the descriptions of the multimodal model training method or data processing method embodiments.

[0274] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described multimodal model training method or data processing method.

[0275] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the above-described multimodal model training method or data processing method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described multimodal model training method or data processing method.

[0276] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0277] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0278] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0279] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0280] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a multi-modal model, the method comprising: include: Determine the multimodal model to be trained and the corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained with different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained with different modalities; Based on multiple model training devices, the models to be trained in each modality, and / or the training dataset, determine the model parallel training strategy corresponding to the models to be trained in each modality, wherein the model parallel training strategies corresponding to the models to be trained in each modality are different. Based on the training dataset and the parallel training strategy corresponding to the model to be trained for each modality, the multiple model training devices are used to train the model to be trained for each modality in parallel to obtain the trained multimodal model. 2.The multi-modal model training method of claim 1, wherein, The step of determining the parallel training strategy for each modality's training model based on multiple model training devices, the models to be trained in each modality, and / or the training dataset includes: Based on the modality of the model to be trained for each modality, determine the model training strategy set corresponding to the model to be trained for each modality, and determine the training data corresponding to the model to be trained for each modality from the training dataset. Among them, the modes of the training models to be trained in each mode are different, the training data corresponding to the training models to be trained in each mode are different, and the initial model training strategies corresponding to the training models to be trained in each mode are also different. Multiple model training devices are identified, and based on the device information of the multiple model training devices, the model information of the model to be trained in each modality, and / or the data information of the training data corresponding to the model to be trained in each modality, a parallel training strategy for the model to be trained in each modality is determined from the model training strategy set corresponding to the model to be trained in each modality.

3. The multimodal model training method according to claim 2, characterized in that, The multiple different modalities of the models to be trained include a visual model to be trained and a language model to be trained. The step of determining the initial model training strategy corresponding to the model to be trained for each modality based on the modality of the model to be trained for each modality, and determining the training data corresponding to the model to be trained for each modality from the training dataset, includes: Based on the visual modality of the visual model to be trained, determine the visual model training strategy set corresponding to the visual model to be trained, and determine the visual training data corresponding to the visual model to be trained from the training dataset; and Based on the language modality of the language model to be trained, determine the language model training strategy set corresponding to the language model to be trained, and determine the language training data corresponding to the language model to be trained from the training dataset.

4. The multimodal model training method according to claim 3, characterized in that, The visual model training strategy set includes data parallel training strategy, tensor parallel training strategy, pipeline parallel training strategy, and context parallel training strategy; the language model training strategy set includes data parallel training strategy, tensor parallel training strategy, and pipeline parallel training strategy. The step of determining the parallel training strategy for each modality's training model from the model training strategy set corresponding to each modality's training model, based on the device information of the multiple model training devices, the model information of each modality's training model, and / or the data information of the training data corresponding to each modality's training model, includes: Based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and / or the data information of the visual training data, at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; and Based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained.

5. The multimodal model training method according to claim 1, characterized in that, The multiple different modalities of the models to be trained include a visual model to be trained and a language model to be trained. The step of determining the parallel training strategy for each modality's training model based on multiple model training devices, the models to be trained in each modality, and / or the training dataset includes: Based on multiple model training devices, the visual model to be trained, and / or the visual training data corresponding to the visual model to be trained in the training dataset, a parallel training strategy for the visual model to be trained is determined; and Based on multiple model training devices, the language model to be trained, and / or the language training data corresponding to the language model to be trained in the training dataset, determine the parallel training strategy for the language model to be trained.

6. The multimodal model training method according to claim 5, characterized in that, The step of determining the parallel training strategy for the visual model corresponding to the visual model to be trained based on multiple model training devices, the visual model to be trained, and / or the visual training data corresponding to the visual model to be trained in the training dataset includes: Based on the visual modality of the visual model to be trained, determine the visual model training strategy set corresponding to the visual model to be trained, and determine the visual training data corresponding to the visual model to be trained from the training dataset. Based on the device information of the multiple model training devices, the model information of the visual model to be trained, and / or the data information of the visual training data, at least one of the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained. Accordingly, determining the parallel training strategy for the language model corresponding to the language model to be trained based on multiple model training devices, the language model to be trained, and / or the language training data corresponding to the language model to be trained in the training dataset includes: Based on the language modality of the language model to be trained, determine the language model training strategy set corresponding to the language model to be trained, and determine the language training data corresponding to the language model to be trained from the training dataset; Based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy is determined as the language model parallel training strategy corresponding to the language model to be trained.

7. The multimodal model training method according to claim 4 or 6, characterized in that, The step of determining at least one of the following—the data parallel training strategy, the tensor parallel training strategy, the pipeline parallel training strategy, and the context parallel training strategy—as the visual model parallel training strategy corresponding to the visual model to be trained, based on the device information of the plurality of model training devices, the model information of the visual model to be trained, and the data information of the visual training data, includes: If, based on the device information of the multiple model training devices, it is determined that the computing resources of the multiple model training devices exceed a preset resource threshold, then the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy are determined as the visual model parallel training strategy corresponding to the visual model to be trained; or If, based on the model information of the visual model to be trained, the model depth of the model to be trained is determined to be greater than a preset depth threshold, then the pipelined parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or If, based on the data information of the visual training data, it is determined that the data context of the visual training data is greater than a preset context threshold, the context-parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or If, based on the model information of the visual model to be trained, it is determined that the model parameters of the model to be trained are greater than a preset parameter threshold, then the tensor parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained; or If, based on the data information of the visual training data, the amount of visual training data is determined to be greater than a preset data amount threshold, the data parallel training strategy is determined as the visual model parallel training strategy corresponding to the visual model to be trained.

8. The multimodal model training method according to claim 4 or 6, characterized in that, The step of determining at least one of the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy as the language model parallel training strategy corresponding to the language model to be trained, based on the device information of the multiple model training devices, the model information of the language model to be trained, and the data information of the language training data, includes: If, based on the device information of the multiple model training devices, it is determined that the computing resources of the multiple model training devices are greater than a preset resource threshold, the data parallel training strategy, the tensor parallel training strategy, and the pipeline parallel training strategy are determined as the language model parallel training strategy corresponding to the language model to be trained.

9. The multimodal model training method according to claim 5, characterized in that, The step of training the training models of each modality in parallel using the multiple model training devices according to the training dataset and the model parallel training strategy corresponding to the training models of each modality, to obtain the trained multimodal model, includes: Based on the visual training data and the visual model parallel training strategy, the visual model to be trained is trained in parallel using the multiple model training devices to obtain the visual training data features of the visual training data. Based on the language training data, the visual training data features, and the language model parallel training strategy, the language model to be trained is trained in parallel using the multiple model training devices to obtain the language training data features of the language training data. Based on the visual training data features and the language training data features, the visual model to be trained and the language model to be trained are trained to obtain the trained multimodal model.

10. The multimodal model training method according to claim 5, characterized in that, The step of training the training models of each modality in parallel using the multiple model training devices according to the training dataset and the model parallel training strategy corresponding to the training models of each modality, to obtain the trained multimodal model, includes: Based on the language training data and the language model parallel training strategy, the language model to be trained is trained in parallel using the multiple model training devices to obtain the language training data features of the language training data. Based on the visual training data, the language training data features, and the visual model parallel training strategy, the visual model to be trained is trained in parallel using the multiple model training devices to obtain the visual training data features of the visual training data. Based on the visual training data features and the language training data features, the visual model to be trained and the language model to be trained are trained to obtain the trained multimodal model.

11. A data processing method, characterized in that, include: A multimodal task processing request is determined, wherein the multimodal task processing request carries visual data or language data; The visual data or the language data is input into the multimodal model to obtain the language data corresponding to the visual data, or the visual data corresponding to the language data. The multimodal model is obtained by training using the multimodal model training method described in any one of claims 1-10.

12. A multimodal model training device, characterized in that, include: The training data determination module is configured to determine the multimodal model to be trained and the corresponding training dataset, wherein the multimodal model to be trained includes multiple models to be trained with different modalities, and the training dataset includes multiple sample pairs, each sample pair consisting of training data corresponding to the multiple models to be trained with different modalities. The training strategy determination module is configured to determine the parallel training strategy corresponding to the model to be trained in each modality based on multiple model training devices, the model to be trained in each modality, and / or the training dataset, wherein the parallel training strategies corresponding to the model to be trained in each modality are different. The model training module is configured to perform parallel training on the models to be trained in each modality using the multiple model training devices according to the training dataset and the model parallel training strategy corresponding to the models to be trained in each modality, so as to obtain the trained multimodal model.

13. A data processing apparatus, characterized in that, include: The request determination module is configured to determine a multimodal task processing request, wherein the multimodal task processing request carries visual data or language data. The data acquisition module is configured to input the visual data or the language data into the multimodal model to obtain language data corresponding to the visual data, or visual data corresponding to the language data. The multimodal model is obtained by training using the multimodal model training method described in any one of claims 1-10.

14. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium storing a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 11.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 11.

Images