Model training method, device, storage medium and computer equipment

Abstract

The embodiments of the present application provide a model training method, apparatus, storage medium and computer equipment, including: obtaining the aggregation parameters of the task visual adapter and the language model in the multimodal language model sent by the designated device, updating the local language model according to the language model aggregation parameters, and screening out and updating the target aggregation parameters of the local target task visual adapter. Then, a task hybrid adapter is constructed to replace the target task visual adapter, which includes a probability predictor, each task adapter and a cross-task adapter to be trained. Finally, the probability predictor, the target task visual adapter and the language model in the task hybrid adapter are trained using a local sample data set to obtain a trained multimodal language model. By training the probability predictor and the cross-task adapter to adapt to different tasks, the multimodal model can learn the patterns and features unique to each task, thereby improving the model performance.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, device, storage medium and computer equipment. Background Art

[0002] With the rapid advancement and widespread application of artificial intelligence (AI) technology, AI has gradually developed into a multimodal model that can perform multiple tasks, such as a multimodal large language model that can perform image description tasks, visual question answering tasks, and visual localization tasks.

[0003] In related technologies, since the diversity of tasks will lead to task heterogeneity, in the model training scenario based on federated learning, the traditional Federated Instruction Tuning (FedIT) method cannot fully solve this task heterogeneity, making it impossible for the multimodal model to learn the unique patterns and characteristics of each task, resulting in a decrease in model performance. Therefore, related technologies urgently need to propose a model training method to solve the above technical problems. Summary of the invention

[0004] The main purpose of this application is to provide a model training method, device, storage medium and computer equipment, which can improve the accuracy and consistency of the appearance of an object from different viewing angles.

[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0006] Obtaining aggregation parameters of a task visual adapter corresponding to each task in a multimodal language model to be trained sent by a designated computer device, and aggregation parameters of the language model;

[0007] Update the local language model according to the aggregation parameters of the language model;

[0008] Filtering the target aggregation parameters of the local target task visual adapter corresponding to the target task from the aggregation parameters of each of the task visual adapters, and updating the local target task visual adapter according to the target aggregation parameters;

[0009] Based on the aggregated parameters of the task vision adapter corresponding to each task, construct a task hybrid adapter, wherein the task hybrid adapter includes a probability predictor, a target task vision adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed on each of the other task adapters;

[0010] The probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained using a local sample data set to obtain a trained multimodal language model.

[0011] In a second aspect, an embodiment of the present application provides a model training method, comprising:

[0012] receiving preliminary training parameters of the task vision adapter and the language model of the corresponding task sent by each computer device;

[0013] Aggregate the initial training parameters of the task vision adapters for the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task;

[0014] Performing parameter aggregation on the preliminary training parameters of the plurality of language models to obtain aggregated parameters of the language model corresponding to each of the computer devices;

[0015] The aggregated parameters of the task vision adapter corresponding to each task and the corresponding aggregated parameters of the language model are sent to each of the computer devices.

[0016] In a third aspect, an embodiment of the present application provides a model training device, comprising:

[0017] An acquisition unit, used for acquiring the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device, and the aggregation parameters of the language model;

[0018] An updating unit, configured to update a local language model according to the aggregation parameters of the language model;

[0019] A screening unit, configured to screen out a target aggregation parameter of a local target task visual adapter corresponding to a target task from the aggregation parameters of each of the task visual adapters, and update the local target task visual adapter according to the target aggregation parameter;

[0020] A construction unit, configured to construct a task hybrid adapter based on aggregated parameters of a task visual adapter corresponding to each task, wherein the task hybrid adapter includes a probability predictor, a target task visual adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed on each of the other task adapters;

[0021] A training unit is used to train the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained through a local sample data set to obtain a trained multimodal language model.

[0022] In a fourth aspect, an embodiment of the present application provides a model training device, including:

[0023] A receiving unit, used for receiving preliminary training parameters of a task vision adapter and a language model of a corresponding task sent by each computer device;

[0024] A first parameter aggregation unit is used to aggregate the initial training parameters of the task vision adapters of the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task;

[0025] A second parameter aggregation unit, configured to aggregate the preliminary training parameters of the plurality of language models to obtain an aggregated parameter of the language model corresponding to each of the computer devices;

[0026] A sending unit is used to send the aggregation parameters of the task vision adapter corresponding to each task and the corresponding aggregation parameters of the language model to each of the computer devices.

[0027] In a fifth aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute any of the above model training methods.

[0028] In a sixth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above model training methods when executing the computer program.

[0029] In an embodiment of the present application, the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device and the aggregation parameters of the language model are obtained; the local language model is updated according to the aggregation parameters of the language model; the target aggregation parameters of the target task visual adapter corresponding to the local target task are screened out from the aggregation parameters of each of the task visual adapters, and the local target task visual adapter is updated according to the target aggregation parameters; based on the aggregation parameters of the task visual adapter corresponding to each task, a task hybrid adapter is constructed, and the target task visual adapter is replaced with the task hybrid adapter, and the task hybrid adapter It includes a probability predictor, a target task visual adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed in each of the other task adapters; the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained through a local sample data set to obtain a trained multimodal language model. Compared with the related art, which cannot fully solve the task heterogeneity between different tasks, the embodiment of the present application adapts to different tasks by training the probability predictor and the cross-task adapter, so that the multimodal model can learn the unique patterns and features of each task, thereby improving the model performance.

[0030] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of a scenario of a model training system provided in an embodiment of the present application.

[0033] Figure 2 A flowchart of the model training method provided in an embodiment of the present application.

[0034] Figure 3 A schematic diagram of the structure of the first stage training of the multimodal language model provided in an embodiment of the present application.

[0035] Figure 4A schematic diagram of the structure of the second stage training of the multimodal language model provided in an embodiment of the present application.

[0036] Figure 5 A schematic diagram of the structure of a task mixing adapter provided in an embodiment of the present application.

[0037] Figure 6 Another flowchart of the model training method provided in an embodiment of the present application.

[0038] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application.

[0039] Figure 8 Another structural schematic diagram of the model training device provided in an embodiment of the present application.

[0040] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0042] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0043] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0044] Artificial Intelligence (AI) is an important part of the intelligent discipline, attempting to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial Intelligence is a very broad science, including robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.

[0045] LLM model: Large Language Model, is a model based on machine learning and natural language processing technology. LLM learns the ability of human language understanding and generation by training a large amount of text data. Its core idea is to use large-scale unsupervised training to automatically discover patterns and language structures in natural language, and then simulate the human language cognition and generation process to a certain extent.

[0046] The working principle of LLM model includes:

[0047] 1. Data training, learning from massive amounts of text data, which covers almost all the content on the Internet for a long period of time. Through unsupervised learning AI algorithms, the model can discover the rules and patterns in the data by itself, such as learning the relationship between words, semantic concepts, etc.

[0048] 2. Model architecture, usually based on the Transformer architecture, consists of multiple layers, including feedforward layers, embedding layers, and attention layers. Among them, the attention mechanism is a key part. For example, the self-attention mechanism can weigh the importance of different tags in the sequence, helping the model capture the dependencies and semantic structures in the text, so as to better understand and generate text.

[0049] Characteristics of the LLM model include:

[0050] 1. Powerful language comprehension and generation capabilities. Compared with traditional NLP models, LLM can more accurately understand the meaning of natural text and generate coherent and logical text content, and can even demonstrate certain logical thinking and reasoning abilities.

[0051] 2. Large-scale parameters. With billions or even trillions of parameters, a large number of parameters enables the model to learn richer language knowledge and semantic information, thereby improving the performance and effect of language processing, but it also leads to the need for huge computing resources and data support for model training and storage.

[0052] The MLLM model, or Multimodal Large Language Model, is a model that integrates information from multiple modalities (such as text, images, audio, etc.) and uses the language model's powerful text processing capabilities to handle multimodal tasks. Unlike traditional LLMs that only process text, MLLM can integrate information from different modalities to better understand and generate text related to multimodal content.

[0053] The working principle and architectural features of the MLLM model include:

[0054] 1. Multimodal information fusion, MLLM usually contains multiple different modal processing modules. Taking the multimodality of text and images as an example, it will have an image encoder to extract image features, which may be abstract representations of information such as the shape, color, and position of objects in the image. At the same time, there are corresponding text processing modules for the text part, such as the word embedding layer, the language model part of the Transformer architecture, etc. Then, the image features and text features are integrated together through specific fusion strategies (such as splicing, attention mechanism fusion, etc.), so that the model can use information from both modalities at the same time.

[0055] For example, in a visual question answering task, the image encoder converts the input question image into a feature vector, and the language model converts the question text into a text feature vector. These two vectors are then fused, and the model generates the answer text based on the fused features.

[0056] 2. Multimodal processing based on language models, using the architecture and training methods of language models to process multimodal information. The language model part has learned a lot of text knowledge and language structure during the pre-training process. In MLLM, it will use the fused multimodal information as input, understand the multimodal content through the application of text knowledge, and generate corresponding text output. For example, in the image subtitle generation task, the language model will generate complete image subtitles based on the fused image and the initial text prompt (such as "Describe this image").

[0057] Pre-training and fine-tuning strategies for MLLM models:

[0058] The training of MLLM usually includes two stages: pre-training and fine-tuning. In the pre-training stage, the model will perform unsupervised learning on large-scale multimodal data to learn the associations between different modalities and how text is expressed in multimodal scenarios. For example, by comparing images and corresponding text descriptions, it learns how to align image features with appropriate text features. In the fine-tuning stage, for specific multimodal tasks (such as visual question answering, image captioning, etc.), the model is further trained using task-specific datasets to optimize the model's performance on specific tasks.

[0059] Current instruction tuning methods are usually implemented through a centralized learning paradigm, where a central party collects a large amount of data to train the model, which may lead to privacy and security issues. Federated learning is currently the main framework for distributed training models while maintaining data security and privacy. A federated learning algorithm, FedAvg, is proposed, which aggregates model gradients from local clients to a central server without data sharing. Afterwards, federated learning is also successfully used for instruction tuning of pre-trained models, called federated instruction tuning (FedIT). These frameworks can effectively utilize local private instruction tracking data. Although existing FedIT methods have made important progress, they only consider the case where different clients perform different instruction-based natural language processing (NLP) tasks, and there is little exploration in the multimodal field. In this case, an attempt is made to transfer the success of FedIT to multimodal tasks. Therefore, a new federated multimodal instruction tuning task (FedMIT) is explored. Compared with traditional FedIT, FedMIT focuses on clients with different multimodal instruction tuning tasks (e.g., visual question answering and image captioning). We first distributed different task instruction data to each client, including image datasets for image description (e.g., COCO dataset), visual question answering datasets (Generalized Question Answering, GQA) for visual question answering, and visual localization task datasets (e.g., RefCOCO dataset) for visual localization. Then, we directly applied Shepherd, a representative method in FedIT tasks, to FedMIT tasks. To determine the relative scores of Shepherd on the two tasks, where the relative score is the comparison score between the results of the Shepherd method and the local training results. The results show that traditional FedIT methods perform poorly on FedMIT tasks. Since the diversity of multimodal tasks greatly increases the heterogeneity between clients, traditional FedIT methods cannot adequately address this task heterogeneity.

[0060] In order to solve the above problems, the embodiments of the present application adapt different tasks by training probability predictors and cross-task adapters, so that the multimodal model can learn the unique patterns and features of each task and improve the model performance. Please refer to the following specific embodiments for details.

[0061] See also Figure 1 , Figure 1 A schematic diagram of a scenario of a model training system provided in an embodiment of the present application, which includes a client 140, the Internet 130, a gateway 120, a server 110, etc.

[0062] The client 140 includes but is not limited to pre-configured electronic devices such as desktop computers, laptop computers, tablet computers, etc. In addition, it can be a single device or a collection of multiple devices. The client 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0063] The server 110 refers to a computer system that can provide certain services to the client 140. Compared with the common client 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0064] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the client 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the client 140 must also be sent to the corresponding client 140 through the gateway 120.

[0065] The model training method of the embodiment of the present disclosure can be implemented on the client 140 .

[0066] It should be noted that Figure 1 The scenario diagram of the model training system shown is only an example. The model training system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can know that with the evolution of image processing technology and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0067] In this embodiment, the model training device will be described from the perspective of the model training device, which can be specifically integrated in a client having a storage unit and a microprocessor installed therein and having computing capabilities.

[0068] See also Figure 2 , Figure 2 A flow chart of a model training method provided in an embodiment of the present application. The model training method comprises:

[0069] In step 201, the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device and the aggregation parameters of the language model are obtained.

[0070] Among them, see Figure 3 , Figure 3 A schematic diagram of the structure of the first stage training of the multimodal language model provided in the embodiment of the present application. In the first stage of the training process of the multimodal language model, the multimodal language model MLLM includes a visual encoder , Image Vision Adapter , Task Vision Adapter , text encoder and language model LLM. Visual encoder It is used to extract visual features from the input image, and the text encoder is used to extract instruction features from the input task instructions. Different clients (which can be understood as entities with different data sources or usage scenarios) often have unique data distributions. Image Vision Adapter In order to adapt to the difference in data distribution, visual features that are closely related to the data distribution characteristics of a specific client are extracted from the visual data. In the context of multimodal task processing, different tasks (such as visual question answering, image caption generation, object detection, etc.) have different requirements for visual features. Task Vision Adapter The main role of is to focus on extracting visual information related to a specific task.

[0071] Specifically, through the first stage of training, each client sends the preliminary training parameters of the local task visual adapter and the preliminary training parameters of the language model to the server so that the server can aggregate the preliminary training parameters of the task visual adapter corresponding to each task, and aggregate the preliminary training parameters of the language model, and return the aggregated parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained, and the aggregated parameters of the language model to each client, so that each client can start the second stage of training.

[0072] It can be understood that for a language model that involves a large number of model parameters, the training of the model parameters of the language model can be replaced by training a text adapter to improve the training rate. Therefore, the parameters of the language model mentioned in the embodiments of the present application can be the model parameters of the language model itself, or the model parameters of the text adapter, which is not limited here.

[0073] In step 202, the local language model is updated according to the aggregation parameters of the language model.

[0074] Among them, after the client receives the aggregation parameters of the language model LLM, the model parameters of the language model LLM obtained after the local first-stage training will be updated according to the aggregation parameters of the language model LLM, so as to facilitate the subsequent second-stage training.

[0075] In step 203, the target aggregation parameters of the target task vision adapter corresponding to the local target task are screened out from the aggregation parameters of each of the task vision adapters, and the local target task vision adapter is updated according to the target aggregation parameters.

[0076] Among them, since in the first stage of training, different clients only train the task vision adapters of their corresponding tasks, after receiving the aggregation parameters of the task vision adapter corresponding to each task, it is necessary to filter out the target aggregation parameters of the target task vision adapter of the local corresponding target task, so as to update the local target task vision adapter according to the target aggregation parameters.

[0077] For example, there are three different tasks, namely, task a, task b, and task c. The task corresponding to the local client is task b, that is, the task visual adapter in the local MLLM model of the client is the task visual adapter corresponding to task b. After receiving the aggregation parameters of the task visual adapters corresponding to task a, task b, and task c, the local client will filter out the aggregation parameters corresponding to task b to update the task visual adapter corresponding to the local task b.

[0078] In step 204, based on the aggregated parameters of the task vision adapter corresponding to each task, a task hybrid adapter is constructed, wherein the task hybrid adapter includes a probability predictor, a target task vision adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed on each of the other task adapters.

[0079] Among them, see Figure 4 , Figure 4 A structural diagram of the second stage training of the multimodal language model provided in the embodiment of the present application. Before the second stage training, the structure of the local MLLM model needs to be adjusted, and the original client-local target task visual adapter after the aggregation parameters are updated only for the target task is replaced with a task hybrid adapter including task visual adapters for different tasks, and the task hybrid adapter is used to perform mixed output of different task visual adapters for visual features.

[0080] Specifically, in order to solve the problem of task heterogeneity between different tasks, a cross-task adapter to be trained is constructed on other task adapters corresponding to each other task except the target task corresponding to the client locally, so that the task adapters corresponding to different tasks with task heterogeneity can be adapted through the cross-task adapter to reduce or avoid the problem of task heterogeneity.

[0081] Specifically, the task-mixing adapter includes a probability predictor , Target Task Visual Adapter , each other task corresponds to other task adapters , = 2, ..., T, T is the total number of tasks, For each task, other tasks other than the target task are assigned. for the task, and in each of the other task adapters The cross-task adapter to be trained built on .

[0082] In some implementations, constructing a task hybrid adapter to be trained in a local multimodal language model based on aggregated parameters of a task visual adapter corresponding to each task includes:

[0083] (1) Construct the probability predictor to be trained;

[0084] (2) constructing each other task visual adapter according to the aggregated parameters of the task visual adapter of each other task, and constructing a cross-task adapter to be trained on each of the other task visual adapters;

[0085] (3) Updating the network parameters of each of the cross-task adapters to the target aggregation parameters;

[0086] (4) constructing a task mixture adapter to be trained based on the probability predictor, the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters.

[0087] Among them, see Figure 5 , Figure 5 A schematic diagram of the structure of a task hybrid adapter provided in an embodiment of the present application. Added to client local except target task visual adapter Every other task except the visual adapter To alleviate the heterogeneity of tasks caused by other task visual adapters Visual adapter with local target task The difference between each other task visual adapter The parameters are the aggregate parameters of the task vision adapters of other tasks received locally by the client.

[0088] Specifically, each cross-task adapter built The initial parameters of the target task vision adapter are the aggregation parameters The purpose is to adjust each cross-task adapter through the second stage of training The parameters of the target task visual adapter With each other task visual adapter Adapt between them to reduce or avoid task heterogeneity.

[0089] Among them, the probability predictor Used to predict the probability of selecting and activating each adapter from the T total task-specific adapters, i.e., the hit probability. It has a linear layer that uses the visual features to calculate the normalized weight matrix for voting, resulting in the hit probability of each task visual adapter.

[0090] Among them, the hit probability can be realized by the following formula:

[0091] ;

[0092] in, is the hit probability, For visual features, The function is a commonly used activation function that is used to convert a real number vector into a probability vector representing the probability of each category. To pass the probability predictor Acting on visual features An intermediate result generated, Represents the set of real numbers.

[0093] Finally, according to Figure 5 It can be seen that the final output of the task mixing adapter can refer to the following formula:

[0094] ;

[0095] in, is the task mixture feature representation for each task, is the hit probability of the client-side local target task visual adapter, The task visual feature representation for the target task output by the target task visual adapter, Visual adapters for other tasks for probabilistic prediction computation The hit probability, Vision adapters for other tasks Output of visual adapters for other tasks The corresponding visual feature representations of other tasks, Vision adapters for other tasks Cross-task adapters on For visual features The output supplements the visual feature representation of the task, and finally obtains the fused visual feature representation for the task .

[0096] In step 205, the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained using a local sample data set to obtain a trained multimodal language model.

[0097] Among them, after constructing the task hybrid adapter to be trained, the second stage of training begins, that is, the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained through a local sample data set to obtain the trained multimodal language model corresponding to the client locally.

[0098] In some embodiments, the sample data set includes a plurality of data pairs, each data pair is composed of a sample image, a sample generation instruction, and a text label, and the probability predictor, the target task visual adapter, and the language model in the task hybrid adapter to be trained are trained by using the local sample data set to obtain a trained multimodal language model, including:

[0099] (1) obtaining a first visual feature of a first sample image in a first data pair currently used as training data, wherein the first visual feature includes a plurality of first visual sub-features;

[0100] (2) inputting the first visual feature into the probability predictor to predict the hit probability of each task visual adapter, and obtaining the original output value of each first visual sub-feature in each task dimension, the allocation probability of each task visual adapter, and the allocation ratio of each first visual sub-feature to each task visual adapter during the prediction process;

[0101] (3) Determine the load balancing loss based on the allocation probability of each task vision adapter and the corresponding allocation ratio;

[0102] (4) determining a probability predictor loss based on the original output value of each of the first visual sub-features in each task dimension;

[0103] (5) inputting the first visual feature into the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters, respectively, to obtain a first target task visual feature representation for the target task output by the target task visual adapter, other task visual feature representations of the corresponding other tasks output by each of the other task visual adapters, and supplementary task visual feature representations of the corresponding other tasks output by each of the cross-task adapters;

[0104] (6) determining a task mixed feature representation based on the hit probability of each of the task visual adapters, the first target task visual feature representation, the other task visual feature representation of each of the other tasks, and the corresponding supplementary task visual feature representation;

[0105] (7) determining a first language modeling loss based on the task hybrid feature representation, the first text label in the first data pair, and the first instruction feature of the first sample generation instruction in the first data pair;

[0106] (8) constructing a first total loss according to the load balancing loss, the probability predictor loss and the first language modeling loss, training the router in the task hybrid adapter to be trained, the target task visual adapter and the language model according to the first total loss, and determining the next data pair in the sample data set as training data as the first data pair, returning to execute the step of obtaining the first visual feature of the first sample image in the first data pair currently used as training data, wherein the first visual feature includes a plurality of first visual sub-features, until the number of times each sample data pair in the sample data set is used as training data reaches a first preset number, thereby obtaining a trained multimodal language model.

[0107] The sample data set includes multiple data pairs, each of which consists of a sample image, a sample generation instruction, and a text label.

[0108] Specifically, the form of the data pair can be referred to as follows:

[0109] ;

[0110] in, For the The data set on the client, The number of data pairs on the client is , For data set The Data pairs, , and Represent sample images, sample generation instructions and text labels respectively, That is the The first The sample images of data pairs, That is the The first The sample generation instructions for data pairs, That is the The first The text labels for the data pairs.

[0111] For example, a data pair includes "a photo of a beach scene, which includes golden sand, blue sea, several palm trees, some people playing on the beach, and a sailboat in the distance"; "a detailed caption for the image is generated to describe the people and scenery on the beach"; and "next to a golden beach, the blue sea is sparkling. There are several palm trees swaying in the wind on the beach, some people are playing happily, and sailboats can be seen sailing on the sea in the distance." Then "a photo of a beach scene" is , "Generate a detailed caption for this image describing the people and scenery on the beach" is "Beside a golden beach, the blue sea sparkles. There are a few palm trees swaying in the wind on the beach, some people are playing happily, and in the distance you can see sailboats sailing on the sea." .

[0112] The first sample image in the first data pair currently used as training data is input into the visual encoder. To obtain the first visual feature corresponding to the first sample image , That is, the first data pair, the first visual feature includes a plurality of first visual sub-features of different dimensions.

[0113] Specifically, refer to Figure 5 , after getting the first visual feature After that, the first visual feature is input into the probability predictor to predict the hit probability of each task visual adapter. , and obtain the original output value of each of the first visual sub-features in each task dimension during the prediction process , the probability of assigning each of the task visual adapters , and the allocation ratio of each first visual sub-feature to each task visual adapter . It measures the load balance of the task vision adapter in the process of processing the visual sub-feature allocation, and uses the load balancing loss function to make the allocation more reasonable to improve the overall performance of the model.

[0114] For each task visual adapter, its allocation probability refers to the probability share of the task visual adapter in the entire allocation mechanism based on factors such as the current input visual features. It reflects the possibility that the visual feature is assigned to the adapter for processing. The determination of this probability may involve multiple factors, such as the characteristics of the visual feature itself, the model's learning of different tasks, etc., and is usually obtained through a specific calculation method during the prediction or processing of the model. The allocation ratio refers to the specific proportional relationship of each visual sub-feature to each task visual adapter. It is determined based on various factors (such as the performance of visual sub-features in different task dimensions, the characteristics of different task visual adapters, etc.) through certain calculation rules (such as a certain weighted method or proportional allocation rule), which is used to clarify the distribution of each visual sub-feature between each task visual adapter. The load balancing loss is in the following form:

[0115] ;

[0116] in, Represents the load balancing loss, which is an indicator used to measure the imbalance of distribution between task vision adapters during multimodal task processing. By calculating this loss value, the distribution can be made more balanced during model training to improve the overall performance of the model.

[0117] Assignment probability You can refer to the following formula:

[0118] ;

[0119] in, Indicates Sub-feature representation Probability Predictor After processing, the task vision adapter corresponding to the maximum value is the task vision adapter , the value of the instruction function is 1, otherwise it is 0.

[0120] Allocation ratio You can refer to the following formula:

[0121] ;

[0122] in, Is a routing instruction To Adapter The probability of assigning to the adapter is then obtained by summing the processing results of N samples and dividing by N. The probability ratio.

[0123] Specifically, the probability predictor loss can refer to the following formula:

[0124] ;

[0125] in, is the probability predictor loss, is the raw output value (Logits) obtained by the router, Indicates The sub-features represent the The raw output value of the task vision adapter, is a natural constant.

[0126] Specifically, in Figure 5 In the above, the visual features are not only input into the probability predictor, but also respectively input into the target task visual adapter, each other task visual adapter, and each cross-task adapter, so as to obtain the first target task visual feature representation for the target task output by the target task visual adapter, the other task visual feature representations of the corresponding other tasks output by each other task visual adapter, and the supplementary task visual feature representations of the corresponding other tasks output by each cross-task adapter. Through the final output formula of the task mixing adapter, the hit probability based on each task visual adapter, the first target task visual feature representation, , other task visual feature representations for each other task And the corresponding supplementary task visual feature representation , the determined task mixed feature representation .

[0127] Among them, in the task mixed feature representation Afterwards, a first instruction feature obtained by feature encoding the first text label in the first data pair through the text encoder and the first text label in the first data pair are combined to determine a first language modeling loss.

[0128] Specifically, a first total loss is constructed according to the load balancing loss, the probability predictor loss, and the first language modeling loss. The first total loss is the total loss in the second stage training process. The router, the target task visual adapter, and the language model in the task hybrid adapter to be trained are trained according to the first total loss, and the next data pair in the sample data set as training data is determined as the first data pair, and the step of obtaining the first visual feature of the first sample image in the first data pair currently used as training data, the first visual feature including multiple first visual sub-features, is returned to execute until the number of times each sample data pair in the sample data set is used as training data reaches a first preset number of times, and the trained multimodal language model is obtained. The first preset number here is a predetermined number of training rounds, for example, E rounds. When the number of times each sample data pair in the sample data set is used as training data reaches E rounds, it means that the MLLM model has been trained for E rounds through the entire sample data set.

[0129] The first total loss can be referred to the following formula:

[0130] ;

[0131] in, is the first total loss, Modeling loss for the first language, Load balancing loss The weight coefficient of is the probability predictor loss The weight coefficient of .

[0132] In some implementations, determining a task mixed feature representation based on the hit probability of each of the task visual adapters, the first target task visual feature representation, the other task visual feature representation of each of the other tasks, and the corresponding supplementary task visual feature representation includes:

[0133] (1.1) obtaining a product of the first target task visual feature representation and the hit probability of the target task visual adapter to obtain a first calculation result;

[0134] (1.2) calculating the sum of the other task visual feature representation of each of the other tasks and the corresponding supplementary task visual feature representation to obtain the total other task visual feature of each of the other tasks;

[0135] (1.3) calculating the product of the total other task visual features of each of the other tasks and the hit probability of the corresponding task visual adapter to obtain a second calculation result;

[0136] (1.4) Determine a sum of the first calculation result and a plurality of the second calculation results to obtain a task mixed feature representation.

[0137] Among them, refer to the final output formula of the task mixing adapter:

[0138] ;

[0139] The first calculation result is , the total other task visual features of each of the other tasks are , the second calculation result is , the final task mixture feature is expressed as .

[0140] In some implementations, the multimodal language model further includes an image vision adapter, and determining a first language modeling loss based on the task hybrid feature representation, the first text label in the first data pair, and the first instruction feature of the first sample generation instruction in the first data pair includes:

[0141] (1.1) inputting the first visual feature into the image visual adapter to obtain a first image visual feature representation;

[0142] (1.2) fusing the first image visual feature representation and the task mixed feature representation to obtain a first fused visual feature representation;

[0143] (1.3) inputting the first fused visual feature representation and the first instruction feature of the first sample generation instruction in the first data pair into the language model to determine a prediction probability of each first text token in the first text label;

[0144] (1.4) Determine a first language modeling loss based on the predicted probability of each of the first text tokens and the number of tokens of the first text tokens.

[0145] Among them, the first language modeling loss can refer to the following formula:

[0146] ;

[0147] It is the first language modeling loss, which is used to measure the cross-entropy loss formula between the model prediction output and the true label. It is widely used in machine learning, especially in sequence-related tasks such as language models, to evaluate the performance of the model and optimize the model parameters by minimizing this loss, so that the model's prediction results are closer to the actual situation. Q is the number of tokens of the predicted first text token, is a conditional probability, Indicates the first text token in the output All previous outputs, the current output is During the model training process, we hope that the model can accurately predict this probability, and the cross entropy loss is to measure the difference between the probability predicted by the model and the true probability.

[0148] Specifically, the prediction of the language model LLM requires inputting the first fused visual feature representation and the first instruction feature of the first sample generation instruction in the first data pair into the language model to determine the prediction of each first text token in the first text label The prediction probability of ; the first language modeling loss can be determined by the calculation formula of the first language modeling loss.

[0149] The following is an introduction to the first phase of training:

[0150] In some implementations, before obtaining the aggregated parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device and the aggregated parameters of the language model, the method further includes:

[0151] (1) training the image vision adapter, the target task vision adapter of the local corresponding target task, and the language model using a local sample data set to obtain the trained image vision adapter, preliminary training parameters of the target task vision adapter, and preliminary training parameters of the language model;

[0152] (2) Sending preliminary training parameters of the target task vision adapter and preliminary training parameters of the language model to a designated computer device, so that the designated computer device aggregates the preliminary training parameters of the task vision adapter corresponding to the task sent by each computer device to obtain aggregated parameters of each task vision adapter, and aggregates the initial training parameters of each language model to obtain aggregated parameters of the language model corresponding to each computer device, and returns the aggregated network parameters of each task vision adapter and the aggregated parameters of the language model corresponding to each computer device to each computer device.

[0153] Among them, see Figure 3 In the first stage of training, the client's local sample data set is used to train the image vision adapter included in the multimodal language model MLLM. , the task vision adapter corresponding to the target task And the language model LLM.

[0154] Specifically, after training to obtain the trained image vision adapter, the preliminary training parameters of the target task vision adapter and the preliminary training parameters of the language model, it is necessary to send the preliminary training parameters of the target task vision adapter and the preliminary training parameters of the language model to a designated computer device, i.e., a server, so as to perform parameter aggregation on the preliminary training parameters of the task vision adapter of the corresponding task sent by each computer device through the designated computer device to obtain the aggregated parameters of each task vision adapter, and perform parameter aggregation on the first training parameters of each of the language models to obtain the aggregated parameters of the language model corresponding to each computer device, and return the aggregated network parameters of each task vision adapter and the aggregated parameters of the language model corresponding to each computer device to each computer device.

[0155] In some embodiments, the sample data set includes a plurality of data pairs, each data pair is composed of a sample image, a sample generation instruction, and a text label, and the image visual adapter, the local target task visual adapter corresponding to the target task, and the language model are trained by the local sample data set to obtain the trained image visual adapter, the preliminary training parameters of the target task visual adapter, and the preliminary training parameters of the language model, including:

[0156] (1.1) obtaining a second visual feature of a second sample image in a second data pair currently used as training data;

[0157] (1.2) inputting the second visual feature into the image vision adapter and the target task vision adapter respectively, to obtain a second image visual feature representation output by the image vision adapter and a second target task visual feature representation output by the target task vision adapter;

[0158] (1.3) constructing a feature difference loss based on the second image visual feature representation and the second target task visual feature representation;

[0159] (1.4) fusing the second image visual feature representation and the second target task visual feature representation, and inputting the fused second fused visual feature representation and the second instruction feature of the second model training instruction in the second data pair into the language model to determine the prediction probability of each second text token of the second text label in the second data pair;

[0160] (1.5) determining a second language modeling loss based on the predicted probability of each of the second text tokens and the number of tokens of the second text tokens;

[0161] (1.6) Based on the constructed feature difference loss and the second language modeling loss, a second total loss is constructed, and according to the second total loss, the image visual adapter, the target task visual adapter of the local corresponding target task, and the language model are trained, and the next data pair in the sample data set as training data is determined as the second data pair, and the step of obtaining the second visual feature of the second sample image in the second data pair currently used as training data is returned to execute until the number of times each sample data pair in the sample data set is used as training data reaches a second preset number, thereby obtaining the trained image visual adapter, the preliminary training parameters of the target task visual adapter, and the preliminary training parameters of the language model.

[0162] Among them, according to Figure 3 The structure of the multimodal language model MLLM, the second sample image in the second data pair currently used as training data needs to be input to the visual encoder To obtain the second visual feature of the second sample image ; The second visual feature Input to the image vision adapter and the target task visual adapter , get the image vision adapter The output second image visual feature representation , and the target task visual adapter Output of the second target task visual feature representation ;

[0163] Specifically, the feature difference loss can refer to the following formula:

[0164] ;

[0165] in, That is the feature difference loss, the formula The target task visual adapter Output of the second target task visual feature representation , Visual feature representation for the second target task The transpose of The second image visual feature representation , is the Frobenius norm.

[0166] The calculation formula for the second language modeling loss is as follows:

[0167] ;

[0168] Among them, the calculation method of the second language modeling loss is similar to the calculation method of the first language modeling loss. The only difference is that the first language modeling loss is calculated for the first data pair currently used as training data in the second training stage, and the second language modeling loss is calculated for the second data pair currently used as training data in the first training stage. Therefore, the calculation of the second language modeling loss is not repeated here.

[0169] Specifically, the calculation method of the second total loss can refer to the following formula:

[0170] ;

[0171] in, is the feature difference loss The weight coefficient of . The first stage of training is performed by using each sample data pair in the sample data set as training data until the number of times each sample data pair is used as training data reaches a second preset number of times, where the second preset number of times is a predetermined number of training rounds, for example, R rounds. When the number of times each sample data pair in the sample data set is used as training data reaches R rounds, it means that the MLLM model has been trained for R rounds through the entire sample data set.

[0172] As can be seen from the above, the embodiment of the present application obtains the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device, as well as the aggregation parameters of the language model; updates the local language model according to the aggregation parameters of the language model; filters out the target aggregation parameters of the target task visual adapter corresponding to the local target task from the aggregation parameters of each of the task visual adapters, and updates the local target task visual adapter according to the target aggregation parameters; constructs a task hybrid adapter based on the aggregation parameters of the task visual adapter corresponding to each task, and replaces the target task visual adapter with the task hybrid adapter, and the task hybrid adapter The device includes a probability predictor, a target task visual adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed in each of the other task adapters; the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained through a local sample data set to obtain a trained multimodal language model. Compared with the related art, which cannot fully solve the task heterogeneity between different tasks, the embodiment of the present application adapts to different tasks by training the probability predictor and the cross-task adapter, so that the multimodal model can learn the unique patterns and features of each task, thereby improving the model performance.

[0173] The following will be described from the perspective of a model training device, which can be specifically integrated in a server (ie, a designated computer device) having a storage unit and a microprocessor installed therein and having computing capabilities.

[0174] See also Figure 6 , Figure 6 A flow chart of a model training method provided in an embodiment of the present application. The model training method comprises:

[0175] In step 301, preliminary training parameters of a task vision adapter and a language model of a corresponding task sent by each computer device are received.

[0176] Among them, after the first stage of training for the multimodal language model MLLM, each client will send the preliminary training parameters of the task visual adapter corresponding to the task after the local first stage of training And the initial training parameters of the language model .

[0177] In step 302, the initial training parameters of the task vision adapters of the same task are aggregated to obtain the aggregated parameters of the task vision adapter corresponding to each task.

[0178] Among them, in the first stage training process for the multimodal language model MLLM, there are multiple clients training the same task based on their own local sample data sets. For example, the tasks of client A and client B are both to train the image subtitle generation capability of the model, so the server will obtain the preliminary training parameters of the task visual adapters for the image subtitle generation training tasks of client A and client B. Therefore, in the server, the preliminary training parameters of the task visual adapters of the same task will be aggregated to obtain the aggregated parameters of the task visual adapter corresponding to each task.

[0179] In some implementations, the step of aggregating the initial training parameters of the task vision adapters for the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task includes:

[0180] (1) for each first computer device for the same task, obtaining the number of data pairs of the sample data set in each first computer device;

[0181] (2) Obtaining the sum of the number of data pairs of each of the first computer devices to obtain the total number of data pairs;

[0182] (3) calculating the ratio of the number of data pairs of each of the first computer devices to the total number of data pairs, and obtaining a first weight value of the task vision adapter of each of the first computer devices;

[0183] (4) determining the product of the preliminary training parameters of the task vision adapter of each of the first computer devices and the corresponding weight value to obtain the parameters to be aggregated of the task vision adapter of each of the first computer devices;

[0184] (5) Obtaining a sum of the multiple parameters to be aggregated to obtain an aggregate parameter of a task vision adapter corresponding to each task of the first computer device.

[0185] Among them, after the server collects the preliminary training parameters of the task vision adapters of all clients, the task-aware aggregation method is adopted. The task vision adapter mainly learns the visual features specific to the task and has a general understanding of the task. Therefore, the parameters learned in one client can also be shared by clients of the same task. The specific task adapter parameters of the same task are aggregated using weighted average. The aggregation calculation method of the preliminary training parameters of the task vision adapters sent by different clients corresponding to the same task can refer to the following formula:

[0186] ;

[0187] in, is a set of tasks of type The set of clients yes One of the clients (i.e., the first computer device), For Clients The number of data pairs in the sample dataset. The client currently performing the calculation The number of data pairs in the sample data set (that is, the first computer device currently performing calculations). Represents the client currently performing the calculation The number of data pairs in the sample data set and the task type of the task are The ratio of the total data of the sample data set in all clients to the number of clients, that is, the The first weight value of the task visual adapter, For Clients The task type sent for the task is Initial training parameters of the task vision adapter , through this calculation formula, the parameters to be aggregated of the task vision adapter of each first computer device can be calculated.

[0188] In step 303, parameter aggregation is performed on the preliminary training parameters of the plurality of language models to obtain aggregated parameters of the language model.

[0189] The server aggregates the initial training parameters of the language model received from each client to obtain aggregated parameters of the language model.

[0190] In some implementations, the step of performing parameter aggregation on the preliminary training parameters of the plurality of language models to obtain the aggregated parameters of the language model includes:

[0191] (1) for each computer device, obtaining a Euclidean distance between a preliminary training parameter of a corresponding language model and a preliminary training parameter of a language model of each other computer device;

[0192] (2) sorting the plurality of Euclidean distances in ascending order, and selecting a second computer device whose Euclidean distance is within a preset number from the other computer devices;

[0193] (3) determining a second weight value of a preliminary training parameter of a language model of the current computer device based on the number of data pairs of the sample data set in the current computer device and the number of data pairs of the sample data set in each of the second computer devices;

[0194] (4) determining a proportion of the number of data pairs of each second computer device based on the Euclidean distance corresponding to each second computer device;

[0195] Determine a third weight value of a preliminary training parameter of a language model of each of the second computer devices according to the number of data pairs of the sample data set in each of the second computer devices and the corresponding number ratio;

[0196] (5) Determining an aggregation parameter of the language model based on the second weight value, the preliminary training parameters of the language model of the current computer device, the preliminary training parameters of the language model of each of the second computer devices, and the corresponding third weight value.

[0197] Among them, for the preliminary training parameters of all language models received by the server, compared with the fully weighted aggregation method used in the traditional FedIT method, this scheme expects that the text adapter parameters of each client can benefit from the positive impact to the greatest extent during the aggregation process, while effectively reducing the negative impact. Therefore, an adaptive text adapter aggregation method is proposed. First, the Euclidean distance between the preliminary training parameters of each language model and the preliminary training parameters of the language models sent by other (K-1) clients is calculated. Then, the nearest M points are selected for weighted aggregation based on the distance.

[0198] The Euclidean distance between the initial training parameters of each language model and the initial training parameters of the language models sent by other (K-1) clients can be calculated by referring to the following formula:

[0199] ;

[0200] Specifically, is the Euclidean distance formula, Current client count for the server (i.e. the current computer device) sends the initial training parameters of the language model , For K clients except the current client (that is, every other computer device other than the current computer device) Initial training parameters of the language model sent .

[0201] Calculate the current client With each second computer device After the Euclidean distance between the corresponding preliminary training parameters is determined, the nearest M points are selected according to the distance, that is, the multiple Euclidean distances are sorted in order from small to large, and the second computer device whose Euclidean distance is within the first preset number (M) is selected from other computer devices.

[0202] The calculation method of the second weight value of the preliminary training parameter of the language model of the current computer device may refer to the following formula:

[0203] ;

[0204] in, The closest parameter A collection of M clients, is a collection of M clients One of the clients in the. By calculating the current client The number of data pairs in the sample dataset The number of data pairs of the sample data set in each of the second computer devices as well as The ratio of the sum of the values ​​is used to determine the second weight value of the preliminary training parameter of the language model of the current computer device.

[0205] The calculation method of the proportion of the number of data of each second computer device to the number can refer to the following formula:

[0206] ;

[0207] in, Second computer device The proportion of data to quantity, For the current client With the second computer device corresponds to the Euclidean distance between the initial training parameters, For the current client The reciprocal sum of the Euclidean distances between the preliminary training parameters corresponding to each second computer device is calculated to obtain the quantitative proportion of the number of data pairs of each second computer device.

[0208] Specifically, the calculation method of the third weight value of the preliminary training parameter of the language model of each second computer device can refer to the following formula:

[0209] ;

[0210] in, For the second computer device The number of data pairs of the sample data set in each second computer device is calculated by multiplying the number of data pairs of the sample data set in each second computer device by the corresponding third weight value, and summing the calculated result with the number of data pairs of the sample data set in each second computer device. as well as The ratio of the sum of the values ​​is determined as a third weight value of the preliminary training parameters of the language model of each second computer device.

[0211] Finally, the aggregation parameters of the language model are determined by the following formula:

[0212] ;

[0213] in, is the aggregation parameter of the language model, is a collection of M clients The preliminary training parameters of the language model sent by one of the second computer devices.

[0214] In step 304, the aggregated parameters of the task vision adapter corresponding to each task and the aggregated parameters of the corresponding language model are sent to each of the computer devices.

[0215] Among them, after the server determines the aggregation parameters of the task vision adapter corresponding to each task and the aggregation parameters of the language model corresponding to each computer device, it sends the aggregation parameters of the task vision adapter corresponding to each task and the aggregation parameters of the corresponding language model to each computer device.

[0216] In some implementations, the present application embodiment provides a method for generating text based on a multimodal language model trained by the aforementioned model training method, and the method for generating text is applied to any client after the second stage of training is completed, including:

[0217] Acquire target images and target instructions;

[0218] Inputting the target image into a visual encoder to obtain target visual features corresponding to the target image;

[0219] Inputting the target instruction into a text encoder to obtain a target text feature corresponding to the target instruction;

[0220] Inputting the target visual features into an image vision adapter and a task mixing adapter respectively, obtaining a target image visual feature representation output by the image vision adapter and a target fused visual feature representation output by the task mixing adapter;

[0221] fusing the target image visual feature representation and the target fused visual feature representation to obtain a target feature representation to be input;

[0222] The target feature representation and the target text feature are input into the trained language model to obtain the target text predicted by the language model.

[0223] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0224] In order to facilitate better implementation of the model training method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above model training method. The meanings of the nouns are the same as those in the above model training method, and the specific implementation details can refer to the description in the method embodiment.

[0225] See also Figure 7 , Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, wherein the model training device is applied to a client as a computer device. The model training device may include an acquisition unit 601, an update unit 602, a screening unit 603, a construction unit 604, and a training unit 605, etc.

[0226] An acquisition unit 601 is used to acquire the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device, and the aggregation parameters of the language model;

[0227] An updating unit 602, configured to update a local language model according to the aggregation parameters of the language model;

[0228] A screening unit 603 is used to screen out a target aggregation parameter of a local target task visual adapter corresponding to a target task from the aggregation parameters of each of the task visual adapters, and update the local target task visual adapter according to the target aggregation parameter;

[0229] A construction unit 604 is used to construct a task hybrid adapter based on the aggregated parameters of the task visual adapter corresponding to each task, wherein the task hybrid adapter includes a probability predictor, a target task visual adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed on each of the other task adapters;

[0230] The training unit 605 is used to train the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained through a local sample data set to obtain a trained multimodal language model.

[0231] In some embodiments, the construction unit 604 is used to:

[0232] Construct the probability predictor to be trained;

[0233] According to the aggregated parameters of the task vision adapter of each other task, construct each other task vision adapter, and construct a cross-task adapter to be trained on each of the other task vision adapters;

[0234] Updating the network parameters of each of the cross-task adapters to the target aggregation parameters;

[0235] A task mixture adapter to be trained is constructed according to the probability predictor, the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters.

[0236] In some embodiments, the sample data set includes a plurality of data pairs, each data pair is composed of a sample image, a sample generation instruction and a text label, and the training unit 605 is used to:

[0237] Acquire a first visual feature of a first sample image in a first data pair currently used as training data, wherein the first visual feature includes a plurality of first visual sub-features;

[0238] Inputting the first visual feature into the probability predictor, predicting the hit probability of each task visual adapter, and obtaining the original output value of each first visual sub-feature in each task dimension, the allocation probability of each task visual adapter, and the allocation ratio of each first visual sub-feature to each task visual adapter during the prediction process;

[0239] Determine the load balancing loss based on the allocation probability of each task vision adapter and the corresponding allocation ratio;

[0240] Determining a probability predictor loss based on the raw output value of each of the first visual sub-features in each task dimension;

[0241] Inputting the first visual feature to the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters respectively, to obtain a first target task visual feature representation for the target task output by the target task visual adapter, other task visual feature representations of the other tasks corresponding to each of the other task visual adapters, and supplementary task visual feature representations of the other tasks corresponding to each of the cross-task adapters;

[0242] Determine a task mixed feature representation based on the hit probability of each of the task visual adapters, the first target task visual feature representation, the other task visual feature representation of each of the other tasks, and the corresponding supplementary task visual feature representation;

[0243] determining a first language modeling loss based on the task mixture feature representation, a first text label in the first data pair, and a first instruction feature of a first sample generation instruction in the first data pair;

[0244] A first total loss is constructed according to the load balancing loss, the probability predictor loss and the first language modeling loss, and the router in the task hybrid adapter to be trained, the target task visual adapter and the language model are trained according to the first total loss, and the next data pair in the sample data set as training data is determined as the first data pair, and the step of obtaining the first visual feature of the first sample image in the first data pair currently used as training data, wherein the first visual feature includes multiple first visual sub-features, is returned to execute until the number of times each sample data pair in the sample data set is used as training data reaches a first preset number, thereby obtaining a trained multimodal language model.

[0245] In some embodiments, the training unit 605 is further specifically configured to:

[0246] Obtaining the product of the first target task visual feature representation and the hit probability of the target task visual adapter to obtain a first calculation result;

[0247] Calculating the sum of the other task visual feature representation of each of the other tasks and the corresponding supplementary task visual feature representation to obtain the total other task visual feature of each of the other tasks;

[0248] Calculating the product of the total other task visual features of each of the other tasks and the hit probability of the corresponding task visual adapter to obtain a second calculation result;

[0249] Determine the sum of the first calculation result and a plurality of the second calculation results to obtain a task mixed feature representation.

[0250] In some embodiments, the training unit 605 is further specifically configured to:

[0251] Inputting the first visual feature into the image vision adapter to obtain a first image visual feature representation;

[0252] fusing the first image visual feature representation and the task mixed feature representation to obtain a first fused visual feature representation;

[0253] Inputting the first fused visual feature representation and the first instruction feature of the first sample generation instruction in the first data pair into the language model to determine a prediction probability of predicting each first text token in the first text label;

[0254] A first language modeling loss is determined based on the predicted probability of each of the first text tokens and the number of tokens of the first text tokens.

[0255] In some embodiments, the apparatus is further configured to:

[0256] Training the image vision adapter, the local target task vision adapter corresponding to the target task, and the language model through a local sample data set to obtain the trained image vision adapter, preliminary training parameters of the target task vision adapter, and preliminary training parameters of the language model;

[0257] Send the preliminary training parameters of the target task vision adapter and the preliminary training parameters of the language model to the designated computer device, so that the designated computer device aggregates the preliminary training parameters of the task vision adapter of the corresponding task sent by each computer device to obtain the aggregated parameters of each task vision adapter, and aggregates the parameters of the first training parameters of each language model to obtain the aggregated parameters of the language model corresponding to each computer device, and returns the aggregated network parameters of each task vision adapter and the aggregated parameters of the language model corresponding to each computer device to each computer device.

[0258] In some embodiments, the sample data set includes a plurality of data pairs, each data pair is composed of a sample image, a sample generation instruction and a text label, and the device is further specifically used for:

[0259] Acquire a second visual feature of a second sample image in a second data pair currently used as training data;

[0260] Inputting the second visual feature into the image vision adapter and the target task vision adapter respectively, to obtain a second image visual feature representation output by the image vision adapter and a second target task visual feature representation output by the target task vision adapter;

[0261] constructing a feature difference loss according to the second image visual feature representation and the second target task visual feature representation;

[0262] fusing the second image visual feature representation and the second target task visual feature representation, and inputting the fused second fused visual feature representation and the second instruction feature of the second model training instruction in the second data pair into the language model to determine the prediction probability of each second text token of the second text label in the second data pair;

[0263] determining a second language modeling loss based on the predicted probability of each of the second text tokens and the number of tokens of the second text tokens;

[0264] Based on the constructed feature difference loss and the second language modeling loss, a second total loss is constructed, and according to the second total loss, the image visual adapter, the target task visual adapter of the local corresponding target task, and the language model are trained, and the next data pair in the sample data set as training data is determined as the second data pair, and the step of obtaining the second visual feature of the second sample image in the second data pair currently used as training data is returned to execute until the number of times each sample data pair in the sample data set is used as training data reaches a second preset number, and the trained image visual adapter, preliminary training parameters of the target task visual adapter, and preliminary training parameters of the language model are obtained.

[0265] As can be seen from the above, the embodiment of the present application obtains the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the specified computer device through the acquisition unit 601, and the aggregation parameters of the language model; the updating unit 602 updates the local language model according to the aggregation parameters of the language model; the screening unit 603 screens out the target aggregation parameters of the target task visual adapter corresponding to the local target task from the aggregation parameters of each task visual adapter, and updates the local target task visual adapter according to the target aggregation parameters; the construction unit 604 constructs a task hybrid adapter based on the aggregation parameters of the task visual adapter corresponding to each task, and the task hybrid adapter includes a probability predictor, a target task visual adapter, each other task corresponding to other task adapters, and a cross-task adapter to be trained constructed on each other task adapter; the training unit 605 trains the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained through a local sample data set to obtain a trained multimodal language model. By training the probability predictor and the cross-task adapter to adapt to different tasks, the multimodal model can learn the unique patterns and features of each task and improve the model performance.

[0266] See also Figure 8 , Figure 8 Another structural diagram of a model training device provided in an embodiment of the present application, wherein the model training device is applied to a server as a computer device. The model training device may include a receiving unit 701, a first parameter aggregation unit 702, a second parameter aggregation unit 703, and a sending unit 704, etc.

[0267] A receiving unit 701 is used to receive preliminary training parameters of a task vision adapter and a preliminary training parameter of a language model of a corresponding task sent by each computer device;

[0268] A first parameter aggregation unit 702 is used to aggregate the initial training parameters of the task vision adapters of the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task;

[0269] A second parameter aggregation unit 703, configured to aggregate the preliminary training parameters of the plurality of language models to obtain an aggregated parameter of the language model corresponding to each of the computer devices;

[0270] The sending unit 704 is used to send the aggregation parameters of the task vision adapter corresponding to each task and the corresponding aggregation parameters of the language model to each of the computer devices.

[0271] In some embodiments, the first parameter aggregation unit 702 is configured to:

[0272] For each first computer device for the same task, obtaining the number of data pairs of the sample data set in each first computer device;

[0273] Obtaining the sum of the number of data pairs of each of the first computer devices to obtain the total number of data pairs;

[0274] Calculating the ratio of the number of data pairs of each of the first computer devices to the total number of data pairs, and obtaining a first weight value of the task vision adapter of each of the first computer devices;

[0275] Determine the product of the preliminary training parameters of the task vision adapter of each of the first computer devices and the corresponding weight value to obtain the parameters to be aggregated of the task vision adapter of each of the first computer devices;

[0276] The sum of the plurality of parameters to be aggregated is obtained to obtain an aggregate parameter of the task vision adapter corresponding to each task of the first computer device.

[0277] In some embodiments, the second parameter aggregation unit 703 is used to:

[0278] For each computer device, obtaining a Euclidean distance between a preliminary training parameter of a corresponding language model and a preliminary training parameter of a language model of each other computer device;

[0279] sorting the plurality of Euclidean distances in ascending order, and selecting a second computer device whose Euclidean distance is within a preset number from the other computer devices;

[0280] Determine a second weight value of a preliminary training parameter of a language model of the current computer device based on the number of data pairs of the sample data set in the current computer device and the number of data pairs of the sample data set in each of the second computer devices;

[0281] Determine a proportion of the number of data pairs of each second computer device based on the Euclidean distance corresponding to each second computer device;

[0282] Determine a third weight value of a preliminary training parameter of a language model of each of the second computer devices according to the number of data pairs of the sample data set in each of the second computer devices and the corresponding number ratio;

[0283] An aggregation parameter of the language model is determined based on the second weight value, preliminary training parameters of the language model of the current computer device, preliminary training parameters of the language model of each of the second computer devices, and the corresponding third weight value.

[0284] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0285] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0286] Reference Fig. 9 , Fig. 9 A block diagram of a portion of a computer device for implementing an embodiment of the present disclosure. Computer devices may vary greatly due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (e.g., one or more processors) and a memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 may be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server 600. Furthermore, the central processing unit 622 may be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

[0287] The computer device may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0288] When the computer device is used as a client, the central processor 622 in the computer device can be used to execute the model training method of the embodiment of the present disclosure, for example:

[0289] Obtaining aggregation parameters of a task visual adapter corresponding to each task in a multimodal language model to be trained sent by a designated computer device, and aggregation parameters of the language model;

[0290] Update the local language model according to the aggregation parameters of the language model;

[0291] Filtering the target aggregation parameters of the local target task visual adapter corresponding to the target task from the aggregation parameters of each of the task visual adapters, and updating the local target task visual adapter according to the target aggregation parameters;

[0292] Based on the aggregated parameters of the task vision adapter corresponding to each task, construct a task hybrid adapter, wherein the task hybrid adapter includes a probability predictor, a target task vision adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed on each of the other task adapters;

[0293] The probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained using a local sample data set to obtain a trained multimodal language model.

[0294] When the computer device is used as a server, the central processor 622 in the computer device can be used to execute the model training method of the embodiment of the present disclosure, for example:

[0295] receiving preliminary training parameters of the task vision adapter and the language model of the corresponding task sent by each computer device;

[0296] Aggregate the initial training parameters of the task vision adapters for the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task;

[0297] Performing parameter aggregation on the preliminary training parameters of the plurality of language models to obtain aggregated parameters of the language model corresponding to each of the computer devices;

[0298] The aggregated parameters of the task vision adapter corresponding to each task and the corresponding aggregated parameters of the language model are sent to each of the computer devices.

[0299] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store program code, and the program code is used to execute the model training methods of the aforementioned embodiments.

[0300] The present disclosure also provides a computer program product, which includes a computer program. The processor of the computer device reads and executes the computer program, so that the computer device as a client executes and implements the above-mentioned model training method. For example:

[0301] Obtaining aggregation parameters of a task visual adapter corresponding to each task in a multimodal language model to be trained sent by a designated computer device, and aggregation parameters of the language model;

[0302] Update the local language model according to the aggregation parameters of the language model;

[0303] Filtering the target aggregation parameters of the local target task visual adapter corresponding to the target task from the aggregation parameters of each of the task visual adapters, and updating the local target task visual adapter according to the target aggregation parameters;

[0304] Based on the aggregated parameters of the task vision adapter corresponding to each task, construct a task hybrid adapter, wherein the task hybrid adapter includes a probability predictor, a target task vision adapter, other task adapters corresponding to each other task, and a cross-task adapter to be trained constructed on each of the other task adapters;

[0305] The probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained using a local sample data set to obtain a trained multimodal language model.

[0306] The processor of the computer device reads and executes the computer program, so that the computer device as a server executes and implements the above-mentioned model training method. For example:

[0307] receiving preliminary training parameters of the task vision adapter and the language model of the corresponding task sent by each computer device;

[0308] Aggregate the initial training parameters of the task vision adapters for the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task;

[0309] Performing parameter aggregation on the preliminary training parameters of the plurality of language models to obtain aggregated parameters of the language model corresponding to each of the computer devices;

[0310] The aggregated parameters of the task vision adapter corresponding to each task and the corresponding aggregated parameters of the language model are sent to each of the computer devices.

[0311] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0312] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0313] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0314] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0315] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0316] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0317] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0318] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0319] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0320] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A model training method, characterized in that: include: Obtaining aggregation parameters of a task visual adapter corresponding to each task in a multimodal language model to be trained sent by a designated computer device, and aggregation parameters of the language model; Update the local language model according to the aggregation parameters of the language model; Filtering the target aggregation parameters of the local target task visual adapter corresponding to the target task from the aggregation parameters of each of the task visual adapters, and updating the local target task visual adapter according to the target aggregation parameters; Construct the probability predictor to be trained; According to the aggregated parameters of the task vision adapter of each other task, construct each other task vision adapter, and construct a cross-task adapter to be trained on each of the other task vision adapters; Updating the network parameters of each of the cross-task adapters to the target aggregation parameters; According to the probability predictor, the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters, a task hybrid adapter to be trained is constructed, wherein the task hybrid adapter includes the probability predictor, the target task visual adapter, each other task corresponding to the other task adapter, and the cross-task adapter to be trained constructed on each of the other task adapters; The probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained are trained using a local sample data set to obtain a trained multimodal language model.

2. The model training method according to claim 1, characterized in that: The sample data set includes a plurality of data pairs, each of which is composed of a sample image, a sample generation instruction, and a text label. The probability predictor, the target task visual adapter, and the language model in the task hybrid adapter to be trained are trained using the local sample data set to obtain a trained multimodal language model, including: Acquire a first visual feature of a first sample image in a first data pair currently used as training data, wherein the first visual feature includes a plurality of first visual sub-features; Inputting the first visual feature into the probability predictor, predicting the hit probability of each task visual adapter, and obtaining the original output value of each first visual sub-feature in each task dimension, the allocation probability of each task visual adapter, and the allocation ratio of each first visual sub-feature to each task visual adapter during the prediction process; Determine the load balancing loss based on the allocation probability of each task vision adapter and the corresponding allocation ratio; Determining a probability predictor loss based on the raw output value of each of the first visual sub-features in each task dimension; Inputting the first visual feature to the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters respectively, to obtain a first target task visual feature representation for the target task output by the target task visual adapter, other task visual feature representations of the other tasks corresponding to each of the other task visual adapters, and supplementary task visual feature representations of the other tasks corresponding to each of the cross-task adapters; Determine a task mixed feature representation based on the hit probability of each of the task visual adapters, the first target task visual feature representation, the other task visual feature representation of each of the other tasks, and the corresponding supplementary task visual feature representation; determining a first language modeling loss based on the task mixture feature representation, a first text label in the first data pair, and a first instruction feature of a first sample generation instruction in the first data pair; A first total loss is constructed according to the load balancing loss, the probability predictor loss and the first language modeling loss, and the router in the task hybrid adapter to be trained, the target task visual adapter and the language model are trained according to the first total loss, and the next data pair in the sample data set as training data is determined as the first data pair, and the step of obtaining the first visual feature of the first sample image in the first data pair currently used as training data, wherein the first visual feature includes multiple first visual sub-features, is returned to execute until the number of times each sample data pair in the sample data set is used as training data reaches a first preset number, thereby obtaining a trained multimodal language model.

3. The model training method according to claim 2, characterized in that: The determining of the task mixed feature representation based on the hit probability of each of the task visual adapters, the first target task visual feature representation, the other task visual feature representation of each of the other tasks, and the corresponding supplementary task visual feature representation comprises: Obtaining a product of the first target task visual feature representation and the hit probability of the target task visual adapter to obtain a first calculation result; Calculating the sum of the other task visual feature representation of each of the other tasks and the corresponding supplementary task visual feature representation to obtain a total other task visual feature of each of the other tasks; Calculating the product of the total other task visual features of each of the other tasks and the hit probability of the corresponding task visual adapter to obtain a second calculation result; Determine the sum of the first calculation result and a plurality of the second calculation results to obtain a task mixed feature representation.

4. The model training method according to claim 2, characterized in that: The multimodal language model also includes an image vision adapter, and determining a first language modeling loss based on the task hybrid feature representation, a first text label in the first data pair, and a first instruction feature of a first sample generation instruction in the first data pair includes: Inputting the first visual feature into the image vision adapter to obtain a first image visual feature representation; fusing the first image visual feature representation and the task mixed feature representation to obtain a first fused visual feature representation; Inputting the first fused visual feature representation and the first instruction feature of the first sample generation instruction in the first data pair into the language model, and determining a prediction probability of predicting each first text token in the first text label; A first language modeling loss is determined based on the predicted probability of each of the first text tokens and the number of tokens of the first text tokens.

5. The model training method according to claim 1, characterized in that: Before obtaining the aggregated parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device and the aggregated parameters of the language model, the method further includes: Training the image vision adapter, the local target task vision adapter corresponding to the target task, and the language model through a local sample data set to obtain the trained image vision adapter, preliminary training parameters of the target task vision adapter, and preliminary training parameters of the language model; Send the preliminary training parameters of the target task vision adapter and the preliminary training parameters of the language model to the designated computer device, so that the designated computer device aggregates the preliminary training parameters of the task vision adapter of the corresponding task sent by each computer device to obtain the aggregated parameters of each task vision adapter, and aggregates the parameters of the first training parameters of each language model to obtain the aggregated parameters of the language model corresponding to each computer device, and returns the aggregated network parameters of each task vision adapter and the aggregated parameters of the language model corresponding to each computer device to each computer device.

6. The model training method according to claim 5, characterized in that: The sample data set includes a plurality of data pairs, each of which is composed of a sample image, a sample generation instruction, and a text label. The image visual adapter, the local target task visual adapter corresponding to the target task, and the language model are trained by the local sample data set to obtain the trained image visual adapter, the preliminary training parameters of the target task visual adapter, and the preliminary training parameters of the language model, including: Acquire a second visual feature of a second sample image in a second data pair currently used as training data; Inputting the second visual feature to the image vision adapter and the target task vision adapter respectively, to obtain a second image visual feature representation output by the image vision adapter and a second target task visual feature representation output by the target task vision adapter; constructing a feature difference loss according to the second image visual feature representation and the second target task visual feature representation; fusing the second image visual feature representation and the second target task visual feature representation, and inputting the fused second fused visual feature representation and the second instruction feature of the second model training instruction in the second data pair into the language model to determine the prediction probability of each second text token of the second text label in the second data pair; determining a second language modeling loss based on the predicted probability of each of the second text tokens and the number of tokens of the second text tokens; Based on the constructed feature difference loss and the second language modeling loss, a second total loss is constructed, and according to the second total loss, the image visual adapter, the target task visual adapter of the local corresponding target task, and the language model are trained, and the next data pair in the sample data set as training data is determined as the second data pair, and the step of obtaining the second visual feature of the second sample image in the second data pair currently used as training data is returned to execute until the number of times each sample data pair in the sample data set is used as training data reaches a second preset number, and the trained image visual adapter, preliminary training parameters of the target task visual adapter, and preliminary training parameters of the language model are obtained.

7. A model training method, characterized in that: include: receiving preliminary training parameters of the task vision adapter and the language model of the corresponding task sent by each computer device; Aggregate the initial training parameters of the task vision adapters for the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task; For each computer device, obtaining a Euclidean distance between a preliminary training parameter of a corresponding language model and a preliminary training parameter of a language model of each other computer device; sorting the plurality of Euclidean distances in ascending order, and selecting a second computer device whose Euclidean distance is within a preset number from the other computer devices; Determine a second weight value of a preliminary training parameter of a language model of the current computer device based on the number of data pairs of the sample data set in the current computer device and the number of data pairs of the sample data set in each of the second computer devices; Determine a proportion of the number of data pairs of each second computer device based on the Euclidean distance corresponding to each second computer device; Determine a third weight value of a preliminary training parameter of a language model of each of the second computer devices according to the number of data pairs of the sample data set in each of the second computer devices and the corresponding number ratio; Determine an aggregation parameter of the language model based on the second weight value, the preliminary training parameters of the language model of the current computer device, the preliminary training parameters of the language model of each of the second computer devices, and the corresponding third weight value; The aggregated parameters of the task vision adapter corresponding to each task and the corresponding aggregated parameters of the language model are sent to each of the computer devices.

8. The model training method according to claim 7, characterized in that: The step of aggregating the initial training parameters of the task vision adapters of the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task includes: For each first computer device for the same task, obtaining the number of data pairs of the sample data set in each first computer device; Obtaining the sum of the number of data pairs of each of the first computer devices to obtain the total number of data pairs; Calculating the ratio of the number of data pairs of each of the first computer devices to the total number of data pairs, and obtaining a first weight value of the task vision adapter of each of the first computer devices; Determine the product of the preliminary training parameters of the task vision adapter of each of the first computer devices and the corresponding weight value to obtain the parameters to be aggregated of the task vision adapter of each of the first computer devices; The sum of the plurality of parameters to be aggregated is obtained to obtain an aggregate parameter of the task vision adapter corresponding to each task of the first computer device.

9. A model training device, characterized in that: include: An acquisition unit, used for acquiring the aggregation parameters of the task visual adapter corresponding to each task in the multimodal language model to be trained sent by the designated computer device, and the aggregation parameters of the language model; An updating unit, configured to update a local language model according to the aggregation parameters of the language model; A screening unit, configured to screen out a target aggregation parameter of a local target task visual adapter corresponding to a target task from the aggregation parameters of each of the task visual adapters, and update the local target task visual adapter according to the target aggregation parameter; A construction unit, used to construct a probability predictor to be trained; According to the aggregated parameters of the task vision adapter of each other task, construct each other task vision adapter, and construct a cross-task adapter to be trained on each of the other task vision adapters; Updating the network parameters of each of the cross-task adapters to the target aggregation parameters; According to the probability predictor, the target task visual adapter, each of the other task visual adapters, and each of the cross-task adapters, a task hybrid adapter to be trained is constructed, wherein the task hybrid adapter includes the probability predictor, the target task visual adapter, each other task corresponding to the other task adapter, and the cross-task adapter to be trained constructed on each of the other task adapters; A training unit is used to train the probability predictor, the target task visual adapter and the language model in the task hybrid adapter to be trained through a local sample data set to obtain a trained multimodal language model.

10. A model training device, characterized in that: include: A receiving unit, used for receiving preliminary training parameters of a task vision adapter and a language model of a corresponding task sent by each computer device; A first parameter aggregation unit is used to aggregate the initial training parameters of the task vision adapters of the same task to obtain the aggregated parameters of the task vision adapter corresponding to each task; A second parameter aggregation unit is used to obtain, for each computer device, a Euclidean distance between a preliminary training parameter of a corresponding language model and a preliminary training parameter of a language model of each other computer device; sorting the plurality of Euclidean distances in ascending order, and selecting a second computer device whose Euclidean distance is within a preset number from the other computer devices; Determine a second weight value of a preliminary training parameter of a language model of the current computer device based on the number of data pairs of the sample data set in the current computer device and the number of data pairs of the sample data set in each of the second computer devices; Determine a proportion of the number of data pairs of each second computer device based on the Euclidean distance corresponding to each second computer device; Determine a third weight value of a preliminary training parameter of a language model of each of the second computer devices according to the number of data pairs of the sample data set in each of the second computer devices and the corresponding number ratio; Determine an aggregation parameter of the language model based on the second weight value, the preliminary training parameters of the language model of the current computer device, the preliminary training parameters of the language model of each of the second computer devices, and the corresponding third weight value; A sending unit is used to send the aggregation parameters of the task vision adapter corresponding to each task and the corresponding aggregation parameters of the language model to each of the computer devices.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the model training method described in any one of claims 1 to 6, or to execute the model training method described in any one of claims 7 to 8.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the model training method described in any one of claims 1 to 6, or the model training method described in any one of claims 7 to 8.

Images