Instruction federated fine-tuning system, method and product of a multi-modal large language model

Abstract

The application provides a multi-modal large language model instruction federal fine-tuning system, method and product, and relates to the technical field of electronic information. The system comprises: a server issuing a plurality of task parameters of a current round to a client; the client modulates image tokens using the semantics of local samples to obtain multi-modal representation, and then processes the multi-modal representation using local individualized parameters of the current round to obtain multi-modal individualized representation; and the server uploads the parameters of the current round obtained by fine-tuning the local model based on the representation to obtain individualized parameters of the current round which are left in the local; the server aggregates the fine-tuned parameters of the current round of the same task according to the fine-tuned parameters of the current round uploaded by each client to obtain and issue the next round parameters of the task; after fine-tuning is completed, the client obtains a final local model based on the local final individualized parameters and the final parameters corresponding to the task issued by the server, and uses the final local model for local task execution. The purpose is to improve the generalization performance of collaborative fine-tuning.

Description

Technical Field

[0001] This application relates to the field of electronic information technology, specifically to a multimodal large language model instruction federation fine-tuning system, method, and product. Background Technology

[0002] In recent years, Multimodal Large Language Model (MLLM) and Vision-Language Model (VLM) have typically adopted a structure of "visual encoder + large language model (LLM) + cross-modal connector" to map image features to the language space, thereby enabling capabilities such as visual question answering, image description, and conversational multimodal reasoning.

[0003] As such models are widely deployed in real-world scenarios such as healthcare, education, industrial quality inspection, and personalized assistants, the high-quality multimodal instruction data required for training and fine-tuning is often scattered across different institutions or terminal devices. This presents challenges due to privacy compliance and data silos, making it difficult to centrally aggregate the data for unified training.

[0004] Against this backdrop, federated learning has emerged as an important technological approach. However, compared to traditional distributed learning, it faces more severe heterogeneity challenges, primarily manifested in statistical heterogeneity (non-IID), meaning significant differences in data distribution, label distribution, and domain style among different clients. Especially in real-world terminal scenarios, each client often engages in multiple tasks simultaneously, presenting a complex reality of coexisting task heterogeneity and data heterogeneity. Therefore, how to stably maintain the model's cross-modal alignment capability under these conditions and effectively achieve personalized adaptation between task-conditional representation learning and client distribution has become a critical issue that urgently needs to be addressed. Summary of the Invention

[0005] In view of this, this application provides a federated fine-tuning system for multimodal large language models, aiming to solve or partially solve the problems existing in the background art.

[0006] The first aspect of this application provides an instruction federated fine-tuning system for a multimodal large language model, the system comprising:

[0007] The server sends global parameters for the current round of multiple sentiment analysis tasks to each client;

[0008] Each client utilizes the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples, thereby obtaining a local task-conditional multimodal representation.

[0009] Each client uses its local current-round personalized parameters to process its local task-conditional multimodal representation, thus obtaining its local task-conditional multimodal personalized representation.

[0010] Each client fine-tunes its local multimodal large language model, which has global parameters and personalized parameters for the current task, based on its local task-conditional multimodal representation and local task-conditional multimodal personalized representation. The fine-tuned global parameters for the current task are uploaded to the server, while the fine-tuned personalized parameters are kept locally for use in the next round of instruction federated fine-tuning.

[0011] The server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task based on the global parameters uploaded by each client, and obtains the global parameters of the next round of tasks for multiple sentiment analysis tasks. The server then sends the global parameters of the next round of tasks for multiple sentiment analysis tasks to each client.

[0012] After the instruction federation fine-tuning is completed, each client obtains its local final personalized parameters, and the server obtains the final global parameters of multiple sentiment analysis tasks and sends them to each client.

[0013] Each client utilizes a local multimodal large language model with local final personalized parameters and final task global parameters from the server to perform multiple sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images.

[0014] A second aspect of this application provides a method for instruction federated fine-tuning of a multimodal large language model, the method comprising:

[0015] In response to the server sending global parameters for multiple sentiment analysis tasks to each client, each client uses the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples, thereby obtaining a local task-conditional multimodal representation.

[0016] Each client uses its local current-round personalized parameters to process its local task-conditional multimodal representation, thus obtaining its local task-conditional multimodal personalized representation.

[0017] Each client fine-tunes its local multimodal large language model, which has global parameters and personalized parameters for the current task, based on its local task-conditional multimodal representation and local task-conditional multimodal personalized representation. The fine-tuned global parameters for the current task are uploaded to the server, while the fine-tuned personalized parameters are kept locally for use in the next round of instruction federated fine-tuning.

[0018] In response to the fine-tuned global parameters of the current round of tasks uploaded by each client, the server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task to obtain the global parameters of the next round of tasks for multiple sentiment analysis tasks, and sends the global parameters of the next round of tasks for multiple sentiment analysis tasks to each client.

[0019] After the instruction federation fine-tuning is completed, each client obtains its local final personalized parameters, and the server obtains the final global parameters of multiple sentiment analysis tasks and sends them to each client.

[0020] Each client utilizes a local multimodal large language model with local final personalized parameters and final task global parameters from the server to perform multiple sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images.

[0021] A third aspect of this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when executed by the processor, the computer program implements the steps of the instruction federation fine-tuning method for a multimodal large language model as described in the first aspect of this application.

[0022] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the instruction federation fine-tuning method for a multimodal large language model as described in the first aspect of this application.

[0023] This application provides a federated instruction fine-tuning system for multimodal large language models. The federated multimodal instruction fine-tuning scheme of this system has the following advantages in real-world scenarios where tasks are non-independent and identically distributed (non-IID) and exhibit significant task heterogeneity:

[0024] The client utilizes the semantics of local sentiment analysis task instruction samples to modulate image tokens of local sentiment analysis task image samples, obtaining a local task-conditionalized multimodal representation. This method, through an explicit instruction conditionalization mechanism, ensures stable expression and transfer of task semantics. Furthermore, the client uses local current-round personalized parameters (client-side personalized parameters do not participate in cross-client aggregation) to process the local task-conditionalized multimodal representation, obtaining a local task-conditionalized multimodal personalized representation. This approach can personalize adaptation to differences in client data distribution. Together with the explicit instruction conditionalization mechanism, this method reduces the risk of negative transfer caused by confusion between task and client factors, effectively decoupling task differences and client data distribution differences, and eliminating their mutual interference. Because it effectively decouples task differences and client data distribution differences, eliminating their mutual interference, this instruction federated fine-tuning system adapts to the reality of single-client multi-task coexistence, thereby achieving stable and controllable collaborative fine-tuning and improved generalization performance. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating an instruction federation fine-tuning system for a multimodal large language model, as shown in one embodiment of this application.

[0027] Figure 2 This is another schematic diagram illustrating an instruction federation fine-tuning system for a multimodal large language model, as shown in one embodiment of this application;

[0028] Figure 3 This is a schematic diagram of an instruction conditional task modeling module in an instruction federation fine-tuning system for a multimodal large language model, as shown in one embodiment of this application.

[0029] Figure 4 This is a flowchart illustrating an instruction federation fine-tuning method for a multimodal large language model, as shown in one embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an instruction federation fine-tuning system for a multimodal large language model, as shown in one embodiment of this application. Figure 1 As shown, the system includes:

[0032] The server sends global parameters for the current round of multiple sentiment analysis tasks to each client;

[0033] Each client utilizes the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples, thereby obtaining a local task-conditional multimodal representation.

[0034] Each client uses its local current-round personalized parameters to process its local task-conditional multimodal representation, thus obtaining its local task-conditional multimodal personalized representation.

[0035] Each client fine-tunes its local multimodal large language model, which has global parameters and personalized parameters for the current task, based on its local task-conditional multimodal representation and local task-conditional multimodal personalized representation. The fine-tuned global parameters for the current task are uploaded to the server, while the fine-tuned personalized parameters are kept locally for use in the next round of instruction federated fine-tuning.

[0036] The server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task based on the global parameters uploaded by each client, and obtains the global parameters of the next round of tasks for multiple sentiment analysis tasks. The server then sends the global parameters of the next round of tasks for multiple sentiment analysis tasks to each client.

[0037] After the instruction federation fine-tuning is completed, each client obtains its local final personalized parameters, and the server obtains the final global parameters of multiple sentiment analysis tasks and sends them to each client.

[0038] Each client utilizes a local multimodal large language model with local final personalized parameters and final task global parameters from the server to perform multiple sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images.

[0039] In this embodiment, the application scenario of the instruction federated fine-tuning system for the multimodal large language model provided in this application is first described as follows: for multiple clients, and each client has multiple tasks coexisting in the instruction federated fine-tuning scenario. For ease of description in the subsequent solution, multiple sentiment analysis tasks are predefined here, and these multiple sentiment analysis tasks include at least the following three tasks:

[0040] Task 1: Classify the emotions in the image based on the prompt text; Task 2: Describe the emotional cues in the image based on the prompt text, such as describing the emotional cues of the task in the image (e.g., the little girl in the image is frowning and has tears in her eyes), for emotion classification; Task 3: Locate the object with the target emotion category in the image based on the prompt text.

[0041] It should be understood that this is only a few sentiment analysis tasks listed for the purpose of describing the solution, and more sentiment analysis tasks may be included in real-world scenarios.

[0042] In this embodiment, different clients can perform federated fine-tuning of instructions for different sentiment analysis tasks. For example, client A performs federated fine-tuning of instructions for tasks 1, 2, and 3, client B performs federated fine-tuning of instructions for tasks 2 and 3, and client C performs federated fine-tuning of instructions for tasks 1 and 3. This application uses a "visual encoder + cross-modal alignment connector + language model (LLM)" as the basic backbone network of the multimodal large language model.

[0043] In this embodiment, after initiating the federated fine-tuning of instructions for the multimodal large language model, the server sends the global parameters of the current round of tasks for each of the multiple sentiment analysis tasks to each client.

[0044] For example, if client A needs to perform instruction federation fine-tuning for tasks 1, 2, and 3, client B needs to perform instruction federation fine-tuning for tasks 2 and 3, and client C needs to perform instruction federation fine-tuning for tasks 1 and 3, then the server will send the current round's global parameters for task 1 to clients A and C, send the current round's global parameters for task 2 to clients A and B, and send the current round's global parameters for task 3 to clients A, B, and C.

[0045] In this embodiment, after each client receives the global parameters for the current round of various sentiment analysis tasks issued by the server, it begins to fine-tune its local multimodal large language model.

[0046] Specifically, each client utilizes the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples (preferably using a cross-attention structure) to obtain a local task-conditional multimodal representation. Here, a sentiment analysis task instruction sample is a multimodal instruction sample (I, xy, m), where I is the image (or other visual input), x is the instruction text, y is the target output, and m∈1,...,M are task identifiers (given by the data source annotation). The semantics of the sentiment analysis task instruction sample is the instruction representation T obtained by encoding the instruction text x using an instruction encoder. The image token of the sentiment analysis task image sample is the result of mapping the image features of image I to the token space that the language model can consume. This modulation process aims to make task differences explicitly driven by instruction semantics, thereby obtaining a more controllable and transferable task-aware representation.

[0047] Each client has its own local personalized parameters for its multimodal large language model. These personalized parameters are not involved in cross-client aggregation (i.e., they are not uploaded to the server; they are only used for fine-tuning the local multimodal large language model). After each client obtains its local task-conditionalized multimodal representation, it uses its local personalized parameters to process its local task-conditionalized multimodal representation, resulting in its own local task-conditionalized multimodal personalized representation. The client-side personalized parameters are designed to adapt to differences in client data distribution (such as non-IID data distribution and domain offset), thereby reducing the risk of negative migration caused by the confusion between task factors and client factors. This effectively decouples task differences from client data distribution differences, eliminating their mutual interference.

[0048] After each client obtains its own local task-conditional multimodal personalized representation, each client inputs its local task-conditional multimodal personalized representation, along with the instruction text x from the sentiment analysis task instruction sample corresponding to that multimodal personalized representation, into its current local multimodal large language model for fine-tuning. This current local multimodal large language model refers to the local multimodal large language model that currently possesses both global parameters and personalized parameters for this round of task. After each client fine-tunes its local multimodal large language model for this round, it uploads the fine-tuned global parameters for this round of task to the server, while simultaneously retaining the fine-tuned personalized parameters locally for use in the next round of instruction federated fine-tuning. That is, when a client performs the next round of instruction federated fine-tuning, the local personalized parameters used are these locally retained fine-tuned personalized parameters for this round.

[0049] When the performance of the server's multimodal large language model meets the conditions for ending fine-tuning, the instruction federation fine-tuning is complete. At this point, each client receives its local final personalized parameters, and the server receives the final global parameters for multiple sentiment analysis tasks. The server then sends the final global parameters for the multiple sentiment analysis tasks to the corresponding clients. For example, during the entire instruction federation fine-tuning process, client A performs instruction federation fine-tuning for tasks 1, 2, and 3; client B performs instruction federation fine-tuning for tasks 2 and 3; and client C performs instruction federation fine-tuning for tasks 1 and 3. The server then sends the final global parameters for task 1 to clients A and C, the final global parameters for task 2 to clients A and B, and the final global parameters for task 3 to clients A, B, and C.

[0050] Finally, each client will utilize its own final local multimodal language model to execute multiple sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images. The client's final local multimodal language model refers to a model containing both local final personalized parameters and final global parameters from the server for each sentiment analysis task.

[0051] The instruction federated fine-tuning system for the multimodal large language model provided in this application has the following advantages in real-world scenarios with non-independent identically distributed (non-IID) tasks and significant task heterogeneity: The client utilizes the semantics of local sentiment analysis task instruction samples to modulate image tokens of local sentiment analysis task image samples, obtaining a local task-conditionalized multimodal representation. This method, through an explicit instruction conditionalization mechanism, ensures stable expression and transfer of task semantics. Furthermore, the client uses local current-round personalized parameters (which do not participate in cross-client aggregation) to process the local task-conditionalized multimodal representation, obtaining a local task-conditionalized multimodal personalized representation. This approach allows for personalized adaptation to differences in client data distribution. Combined with the explicit instruction conditionalization mechanism, this reduces the risk of negative transfer caused by confusion between task and client factors, effectively decoupling task differences and client data distribution differences, and eliminating mutual interference between the two. Because it effectively decouples task differences from client data distribution differences, eliminating mutual interference, this instruction federation fine-tuning system adapts to the reality of single-client, multi-task coexistence, thereby achieving stable and controllable collaborative fine-tuning and generalization performance improvement. This federated multimodal instruction fine-tuning task, characterized by heterogeneous, non-independent, and identically distributed tasks, aims to enable multiple clients to collaboratively improve the understanding and generation capabilities of a multimodal large language model for different instruction tasks under the condition of "data not leaving the client."

[0052] In conjunction with the above embodiments, in one implementation, this application also provides an instruction federation fine-tuning system for a multimodal large language model. In this instruction federation fine-tuning system for a multimodal large language model, before performing the first round of instruction federation fine-tuning, the server sends parameters of frozen cross-modal alignment connectors to each client. These frozen cross-modal alignment connectors are used to map image features to image tokens.

[0053] During each round of instruction federation fine-tuning, each client uses the parameters of the frozen cross-modal alignment connector to map the image features of local sentiment analysis task image samples into image tokens.

[0054] During the process of fine-tuning the local multimodal large language model with global parameters and personalized parameters of the current task based on the local task-conditional multimodal representation and the local task-conditional multimodal personalized representation, each client keeps the parameters of the frozen cross-modal alignment connector unchanged for use in the next round of instruction federated fine-tuning.

[0055] Each client utilizes the frozen cross-modal alignment connector, a local multimodal large language model with local final personalized parameters and final task global parameters from the server, to perform multiple sentiment analysis tasks for local sentiment analysis task instructions and local sentiment analysis task images.

[0056] In this embodiment, to stabilize the alignment channel from the visual to the language space, this application employs a frozen cross-modal alignment connector to map image features to the token space that the language model can consume, obtaining the corresponding image token. This method can stably maintain the visual-language alignment entry point, avoid alignment drift, and thus improve training stability under non-IID conditions. Specifically, in multimodal instruction federated fine-tuning, during the collaborative fine-tuning process of multiple clients, the connector connecting vision and language is prevented from being biased by the differences between clients, and always maintains the ability to accurately and reliably transmit image information to the language model. More specifically, the cross-modal alignment connector is used to project image features onto the input embedding space of the language model, so that the output image token is aligned with the representation of the text token in terms of dimension and semantic level, thereby being consumed by the language model.

[0057] Specifically, before the first round of instruction federation fine-tuning, the server first sends the parameters of the frozen cross-modal alignment connector to each client. The frozen cross-modal alignment connector is used to map image features to image tokens.

[0058] After each client receives the parameters of the frozen cross-modal alignment connector, during each round of instruction federation fine-tuning, each client uses these parameters to map the image features of its local sentiment analysis task image samples into image tokens. In this process, the image features of the images in the sentiment analysis task instruction samples are first extracted using a visual encoder, expressed as: Where V represents image features and I represents the image. These are the parameters for the visual encoder. Then, the client uses these frozen parameters of the cross-modal alignment connector to map the image features of its local sentiment analysis task image samples to the token space that the language model can consume, obtaining the image token, expressed as: ,in, For image tokens, For cross-modal alignment connectors, Freezing parameters for cross-modal alignment connectors can improve training stability under non-IID conditions.

[0059] During the fine-tuning of the local multimodal large language model, which contains both global and personalized parameters for the current task, each client keeps the parameters of the frozen cross-modal alignment connector unchanged for use in the next round of instruction federation fine-tuning. In other words, the parameters of the cross-modal alignment connector remain constant throughout the entire instruction federation fine-tuning process.

[0060] After the final instruction federal fine-tuning is completed, each client uses the frozen cross-modal alignment connector, a local multimodal large language model with local final personalized parameters and final task global parameters from the server, to execute multiple corresponding sentiment analysis tasks on local sentiment analysis task instructions and local sentiment analysis task images.

[0061] In conjunction with the above embodiments, in one implementation, this application also provides an instruction federation fine-tuning system for a multimodal large language model. In this instruction federation fine-tuning system for a multimodal large language model, each client has local parameters for a frozen cross-modal alignment connector, which is used to map image features to image tokens;

[0062] During each round of instruction federation fine-tuning, each client uses the parameters of the local frozen cross-modal alignment connector to map the image features of the local sentiment analysis task image samples into image tokens.

[0063] During the fine-tuning of the local multimodal large language model with global parameters and personalized parameters of the current task based on the local task-conditional multimodal representation and the local task-conditional multimodal personalized representation, each client keeps the parameters of the local frozen cross-modal alignment connector unchanged for the next round of instruction federated fine-tuning.

[0064] Each client utilizes a local frozen cross-modal alignment connector, a local multimodal large language model with local final personalized parameters and final task global parameters from the server, to perform multiple sentiment analysis tasks for local sentiment analysis task instructions and local sentiment analysis task images.

[0065] In this embodiment, in another alternative implementation, each client directly deploys the frozen cross-modal alignment connector, instead of having it distributed by the server.

[0066] Similarly, during each round of instruction federated fine-tuning, each client uses its own locally frozen cross-modal alignment connector parameters to map the image features of its local sentiment analysis task image samples into image tokens. While fine-tuning its local multimodal large language model, which contains both global and personalized parameters for this round of task, each client maintains its locally frozen cross-modal alignment connector parameters unchanged for use in the next round of instruction federated fine-tuning. Each client also utilizes its own locally frozen cross-modal alignment connector, its local final personalized parameters, and the final global task parameters from the server to execute multiple corresponding sentiment analysis tasks for its local sentiment analysis task instructions and local sentiment analysis task images.

[0067] In conjunction with the above embodiments, in one implementation, this application also provides a federated fine-tuning system for multimodal large language models. In this federated fine-tuning system, each client utilizes the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples, obtaining a local task-conditionalized multimodal representation, including:

[0068] Each client obtains its local query vector based on the semantics of the local sentiment analysis task instruction samples;

[0069] Each client obtains its own key vector and value vector based on the image tokens of the local sentiment analysis task image samples.

[0070] Each client uses an attention mechanism to process its local query vector, key vector, and value vector to obtain a local task-conditional multimodal representation.

[0071] In this embodiment, the present application preferably employs a cross-attention structure for image tokenization conditioned on instruction representation T. Task modulation. The specific modulation process is as follows:

[0072] To explicitly model task differences, this application first encodes the instruction text x in the sentiment analysis task instruction sample (such as sample 1) using an instruction encoder to obtain the corresponding instruction representation T. This instruction representation T is the semantics of the sentiment analysis task instruction sample (i.e., sample 1). Each client obtains its local query vector based on the semantics of its own local sentiment analysis task instruction sample. Simultaneously, each client obtains its local key vector and local value vector based on the image token of its own local sentiment analysis task image sample, specifically expressed as follows:

[0073]

[0074] in, To query the projection matrix of Q, Let K be the projection matrix of the key. The projection matrix is ​​for the value U. That is, during this modulation process, the query Q comes from the instruction representation T, the key value K, and U come from the aligned image token. .

[0075] After each client obtains its own local query vector, key vector, and value vector based on its local sentiment analysis task instruction samples, each client uses an attention mechanism to process these vectors. Then, residual and normalization processing is used to obtain the local task-conditional multimodal representation, specifically expressed as follows:

[0076]

[0077]

[0078] Where H represents the number of attention heads in the multi-head attention mechanism, d represents the dimension of each attention head, and Z represents the task-conditional multimodal representation. For image tokens.

[0079] This application achieves more controllable and transferable task-aware representations by explicitly driving task differences through instruction semantics via modulation. Specifically, a dedicated mechanism within the model explicitly incorporates the information of "what task to do now" into representation learning. The model knows which task it is currently performing and adjusts its understanding and representation inputs accordingly. Thus, task representations are no longer ambiguous signals implicit in the data, but rather explicit signals. The core principle is to make task modeling clear. Through a task-aware mechanism, the model knows what task to perform, and updates between different tasks do not interfere with each other, mitigating the negative transfer caused by task heterogeneity. Negative transfer refers to the situation where, if task modeling is unclear, updates to the model by one client may impair updates to the model by different tasks on another client.

[0080] For example, client A's task is to learn "image description," while client B's task is to learn "visual question answering." Through the model's task-aware mechanism, each client's local model knows which task it is learning (i.e., client A's local model knows its task is "image description," and client B's local model knows its task is "visual question answering"). Simultaneously, the server-side global model also knows which task's model update parameters each client is uploading (i.e., client A's uploaded model update parameters are for the "image description" task, and client B's uploaded model update parameters are for the "visual question answering" task). When updating model parameters, the server-side global model ensures that updates for different tasks do not interfere with each other, mitigating the negative transfer caused by task heterogeneity.

[0081] In this embodiment, considering the non-IID differences between clients, such as data distribution and domain offset, this application introduces personalized parameters locally on the client to model the client-specific data distribution, preferably implemented as a lightweight adapter. The client then uses the local personalized parameters for this round to process the local task-conditionalized multimodal representation, obtaining the local task-conditionalized multimodal personalized representation, expressed as:

[0082]

[0083] Where Z represents the multimodal representation of task conditionalization. These are personalized parameters for client k and do not participate in cross-client aggregation to reduce the risk of negative migration caused by the confusion between task factors and client factors. For multimodal personalized representation.

[0084] In conjunction with the above embodiments, in one implementation, this application also provides a federated fine-tuning system for a multimodal large language model. In this federated fine-tuning system, the multiple sentiment analysis tasks include a first sentiment analysis task, and the multimodal training samples for the first sentiment analysis task locally on the first client to the Kth client are non-independent and identically distributed.

[0085] For the first sentiment analysis task, each of the first client to the Kth client uploads the number of multimodal training samples for the first sentiment analysis task locally to the server;

[0086] The server determines the total number of all multimodal training samples for the first sentiment analysis task, and determines the aggregation coefficient for each of the first to K clients based on the ratio of the number of samples uploaded by the first client to the Kth client to the total number.

[0087] Based on the global parameters of the current round of the first sentiment analysis task uploaded by the first client to the Kth client after fine-tuning, the server performs a weighted summation of the global parameters of the current round of the first sentiment analysis task according to the aggregation coefficients of the first client to the Kth client, and then sends the global parameters of the next round of the first sentiment analysis task to the first client to the Kth client.

[0088] In this embodiment, during the instruction federation fine-tuning process, the multimodal training samples (i.e., the sentiment analysis task instruction samples in the aforementioned fine-tuning process) for all clients performing the same sentiment analysis task exhibit a non-independent and identically distributed (i.i.d.) pattern. Non-independent and identically distributed means that the local data volume and / or data type for the same sentiment analysis task differs across clients. For example, 10 clients may all be performing the same task of classifying objects in images, but some clients may contain images primarily of animals like cats and dogs, while others may contain images primarily of various mechanical parts, and still others may contain images primarily of people from different countries. Furthermore, the amount of data in each client may also differ.

[0089] In this embodiment, to improve the aggregation effect of the fine-tuned global parameters of the current round of tasks uploaded by various clients for the same sentiment analysis task, this application provides an optional aggregation method. Since the aggregation here is for the same type of sentiment analysis task, and the aggregation method corresponding to each sentiment analysis task is the same, for ease of description, a first sentiment analysis task is used as an example. Furthermore, during the entire instruction federation fine-tuning process, the clients that will fine-tune the first sentiment analysis task include the first client to the Kth client, as detailed below:

[0090] For this first sentiment analysis task, each of the first to Kth clients uploads its own local number of multimodal training samples for this task to the server. The server then calculates the total number of multimodal training samples for the first sentiment analysis task based on these received samples. Finally, the aggregation coefficient for each of the first to Kth clients is obtained based on the ratio of the number uploaded by each client to this total. For example, the number of multimodal training samples uploaded by the kth client for this first sentiment analysis task is... If the total number of multimodal training samples for the first sentiment analysis task is N, then the aggregation coefficient for the k-th client is determined as follows: k takes values ​​from 1 to K.

[0091] After the server obtains the aggregation coefficients for clients 1 through K, it performs a weighted sum of all the fine-tuned global parameters for the current round of the first sentiment analysis task uploaded by clients 1 through K, based on their respective aggregation coefficients. This sum yields the global parameters for the next round of the first sentiment analysis task. For example, the aggregation coefficient for client k is... Then, when performing a weighted sum of the global parameters for this round of tasks after all the fine-tuning for the first sentiment analysis task from the first client to the Kth client, the global parameters for this round of tasks after fine-tuning for the kth client will be calculated using this aggregation coefficient. Perform aggregation.

[0092] Then, the server sends the next round of global parameters for the first sentiment analysis task to each of the first to Kth clients. Using the same implementation method, the server can perform a weighted summation of all fine-tuned global parameters for the current round of any sentiment analysis task to obtain the next round of global parameters for that task, and then send them to the corresponding clients.

[0093] In conjunction with the above embodiments, in one implementation, this application also provides a multimodal large language model instruction federated fine-tuning system. In this multimodal large language model instruction federated fine-tuning system, a first client among the various clients is used to execute M sentiment analysis tasks out of the plurality of sentiment analysis tasks, and the first client is any one of the various clients;

[0094] Take m as any integer from 1 to M. The first client reads the multimodal training samples for the m-th sentiment analysis task from the local machine according to the m-th sentiment analysis task identifier, so as to obtain the fine-tuned global parameters of the current round of tasks for the m-th sentiment analysis task and upload them to the server. The fine-tuned personalized parameters for the m-th sentiment analysis task are kept locally for the next round of instruction federated fine-tuning.

[0095] After the instruction federation fine-tuning is completed, the first client obtains the final personalized parameters for the m-th sentiment analysis task locally, and the server obtains the final global parameters for the m-th sentiment analysis task and sends them to the first client.

[0096] The first client utilizes a local multimodal large language model for the m-th sentiment analysis task, and executes the m-th sentiment analysis task based on the local sentiment analysis task instructions and the local sentiment analysis task images. The local multimodal large language model for the m-th sentiment analysis task has the final personalized parameters for the m-th sentiment analysis task and the final global parameters for the m-th sentiment analysis task from the server.

[0097] In this embodiment, we will explain how each client performs corresponding instruction federation fine-tuning based on the global parameters of the current round of sentiment analysis tasks. Since each client uses the same instruction federation fine-tuning method for its multiple sentiment analysis tasks, we will use one client as an example. This client is named the first client, and it can be any client among all clients. This first client is defined to execute M sentiment analysis tasks out of all sentiment analysis tasks. Specifically, the global parameters of the current round of sentiment analysis tasks are as follows:

[0098] After the first client receives the global parameters for the current round of M sentiment analysis tasks from the server, it determines the sentiment analysis task identifier for each of the M sentiment analysis tasks. It should be noted that each multimodal training sample used for instruction federated fine-tuning on the client's local machine contains a sentiment analysis task identifier; that is, each multimodal training sample marks which sentiment analysis task it is used for fine-tuning. Then, based on the m-th (where m is any integer from 1 to M) sentiment analysis task identifier, the first client reads the multimodal training sample corresponding to the m-th sentiment analysis task identifier from its local machine. It then uses the read multimodal training sample for the m-th sentiment analysis task identifier for fine-tuning to obtain the fine-tuned global parameters for the m-th sentiment analysis task corresponding to that m-th sentiment analysis task identifier. Finally, it uploads the obtained fine-tuned global parameters for the m-th sentiment analysis task to the server.

[0099] During the fine-tuning process, the personalized parameters of the first client are also fine-tuned locally. The personalized parameters of the first client after fine-tuning for the m-th sentiment analysis task are not uploaded to the server, but are only kept locally for the next round of instruction federated fine-tuning.

[0100] Finally, after the instruction federation fine-tuning is complete, the first client will receive the final personalized parameters for the m-th sentiment analysis task locally, and the server will receive the final global parameters for the m-th sentiment analysis task, which will then be sent to the first client. Finally, the first client utilizes its local multimodal large language model for the m-th sentiment analysis task, and executes the m-th sentiment analysis task based on the local sentiment analysis task instructions and the local sentiment analysis task image. Ultimately, the first client's local multimodal large language model for the m-th sentiment analysis task has the following parameters: the first client's final personalized parameters for the m-th sentiment analysis task locally, and the final global parameters for the m-th sentiment analysis task from the server.

[0101] In conjunction with the above embodiments, in one implementation, this application also provides a multimodal large language model instruction federated fine-tuning system. In this multimodal large language model instruction federated fine-tuning system, the server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task based on the fine-tuned global parameters of the current round of tasks uploaded by each client, to obtain the initial global parameters of the next round of tasks for multiple sentiment analysis tasks;

[0102] The server uses multimodal training samples from multiple local sentiment analysis tasks to update the initial global parameters for the next round of the multiple sentiment analysis tasks, obtains the global parameters for the next round of the multiple sentiment analysis tasks, and sends them to each client.

[0103] In this embodiment, as described above, the server primarily receives the fine-tuned global parameters of the current round of tasks uploaded by each client. Simultaneously, it aggregates all the fine-tuned global parameters of the current round of tasks belonging to the same sentiment analysis task, obtaining the initial global parameters for the next round of tasks for that sentiment analysis task. Through the same implementation method, each sentiment analysis task can obtain its own corresponding initial global parameters for the next round of tasks through aggregation. To further improve the instruction federation fine-tuning effect of this application, after the server obtains the initial global parameters for the next round of tasks for each sentiment analysis task through aggregation, the server further updates the initial global parameters for the next round of tasks for each sentiment analysis task obtained through aggregation using its local multimodal training samples from multiple sentiment analysis tasks. This results in multiple sentiment analysis tasks with better fine-tuning effects having their next-round global parameters for the next round of tasks. These multiple sentiment analysis tasks with better fine-tuning effects are then distributed to the corresponding clients. For example, after the server obtains the initial global parameters for the next round of tasks corresponding to the j-th sentiment analysis task through aggregation... The server then uses its local multimodal training samples of the j-th sentiment analysis task to further update the initial global parameters of the next round of the j-th sentiment analysis task obtained through aggregation. This results in a better-tuned global parameter for the next round of the j-th sentiment analysis task, where j takes any integer from 1 to J, and J is the total number of sentiment analysis tasks.

[0104] In this embodiment, each client, during its local instruction fine-tuning process, completes the fine-tuning of its local multimodal large language model through backpropagation based on its locally calculated instruction fine-tuning loss. The instruction fine-tuning loss is preferably an autoregressive cross-entropy.

[0105]

[0106] Where p(·) represents the probability distribution predicted by the local multimodal large language model under the current parameters; x represents the instruction text; For multimodal personalized representation; The actual token at the t-th position in the target sequence (that is, the t-th word in the output standard answer); This refers to all tokens generated before time step t.

[0107] like Figure 2As shown, the multimodal large language model instruction federation fine-tuning system provided in this application includes a computational link on the client side consisting of "Decoupling Align Module (DecAlign) + Instruction Conditional Task Modeling Module (InstTask) + Client Data Personalization Module (CliData) + Large Language Model (LLM)" and a link on the server side consisting of "Task-Aware Parameter Aggregation Module + Task Routing Parameter Distribution Module".

[0108] The instruction-conditional task modeling module is structurally independent of the decoupled alignment module and the client data personalization module, with a decoupled relationship between alignment, task modeling and client personalization.

[0109] In this model, the decoupled alignment module (i.e., the cross-modal alignment connector) remains frozen during federated training, while the client-side data personalization module is only used for local data distribution adaptation and does not participate in cross-client parameter aggregation. Specifically, on the client side, the image is processed by the visual encoder and the frozen decoupled alignment module (DecAlign) to obtain an image token. The instruction text x is encoded by an instruction encoder to obtain an instruction representation T; the instruction conditionalization task modeling module uses the obtained instruction representation T to conditionally modulate the image token. The task-conditional multimodal representation Z is obtained; the client data personalization module processes the obtained task-conditional multimodal representation Z to obtain the multimodal personalized representation. Ultimately, this will enable multimodal personalized representation. The local multimodal large language model is input to the client with the instruction text x to obtain the output, and the instruction fine-tuning loss and a small number of parameters are calculated based on the output.

[0110] On the server side, InstTask parameters are grouped and aggregated according to task m (or task clusters) and then sent to the client. The client selects the corresponding global parameters based on the sentiment analysis task identifier, realizing a real-world training and inference process of "single client, multiple tasks coexisting".

[0111] During the federated training process, the server sends the global parameters for the sentiment analysis task m during the r-th round of communication. The client updates its multi-task data locally in several steps, updating only the corresponding task's global parameters. And client-side personalized parameters (Custom parameters for client k), parameters of the frozen decoupled alignment module DecAlign (i.e., the cross-modal alignment connector). .

[0112] After this round of fine-tuning, the client uploads the results of the global parameter fine-tuning for each sentiment analysis task to the server. The server, through the task-aware parameter aggregation module TaskAgg, sets the set of clients participating in the fine-tuning of sentiment analysis task m in this round as follows: The corresponding sample size weight is Then the task aggregation expression is:

[0113]

[0114] in, For client k, the fine-tuned global parameters of the sentiment analysis task m are uploaded in round r+1. In this expression, 't' represents the local iteration count. The meaning of this expression is: to aggregate the task-wide parameter fine-tuning results from different clients for the same type of sentiment analysis task; simultaneously, during aggregation, an aggregation weight is set based on the number of samples each client used for that sentiment analysis task, with clients that used more samples for that sentiment analysis having a higher aggregation weight for their uploaded task-wide parameter fine-tuning results. Client-specific parameters. It is not uploaded, but only retained locally for continuous adaptation to the distribution of non-IID data on the client.

[0115] like Figure 3 As shown, the InstTask module of this application, which is used to introduce instruction semantics as conditional information during multimodal representation learning, is used to achieve task-related modeling of visual representations. This module receives instruction representation T from the instruction encoder and aligned image tokens from the decoupled alignment module (DecAlign). Then, based on the instruction representation T for the image token... Conditional modulation is performed to generate task-related multimodal representations. This instruction-conditional task modeling module, InstTask, can be implemented using an attention mechanism, where the instruction representation T is used to generate conditional signals, and the aligned image tokens... As the modulated object, task-related multimodal representations are obtained through conditional computation.

[0116] The multimodal large language model instruction federated fine-tuning system provided in this application reduces federated communication and end-side training overhead by freezing the alignment entry and decoupling task modeling and client personalization, thereby improving the stability and generalization of non-IID multi-task systems. The core of this application lies in: the overall structure of the decoupled multimodal instruction fine-tuning framework, which decouples the alignment, task modeling, and client personalization modules; the frozen decoupled alignment module DecAlign; the instruction-conditionalized task modeling module InstTask, which explicitly modulates the aligned image embedding using instruction embedding to form a task-related multimodal representation that can be aggregated by task; the client data personalization module CliData, which is only updated locally and does not participate in aggregation; and the server-side task-aware aggregation and parameter distribution mechanism, which supports task parameter selection and updating under single-client multi-task coexistence.

[0117] Based on the same inventive concept, this application provides a method for instruction federation fine-tuning of a multimodal large language model, such as... Figure 4 As shown, the instruction federated fine-tuning method for this multimodal large language model includes:

[0118] Step S401: In response to the server sending global parameters for the current round of multiple sentiment analysis tasks to each client, each client uses the semantics of the local sentiment analysis task instruction sample to modulate the image token of the local sentiment analysis task image sample to obtain the local task-conditional multimodal representation.

[0119] Step S402: Each client uses its local current-round personalized parameters to process its local task-conditional multimodal representation to obtain its local task-conditional multimodal personalized representation.

[0120] Step S403: Each client fine-tunes its local multimodal large language model, which has global parameters and personalized parameters for the current task, based on its local task-conditional multimodal representation and local task-conditional multimodal personalized representation. The fine-tuned global parameters for the current task are uploaded to the server, and the fine-tuned personalized parameters for the current task are kept locally for use in the next round of instruction federated fine-tuning.

[0121] Step S404: In response to the fine-tuned global parameters of the current round of tasks uploaded by each client, the server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task to obtain the global parameters of the next round of tasks for multiple sentiment analysis tasks, and sends the global parameters of the next round of tasks for multiple sentiment analysis tasks to each client.

[0122] Step S405: After the instruction federation fine-tuning is completed, each client obtains its local final personalized parameters, and the server obtains the final global parameters of multiple sentiment analysis tasks and sends them to each client.

[0123] Step S406: Each client uses a local multimodal large language model with local final personalized parameters and final task global parameters from the server to perform multiple sentiment analysis tasks for local sentiment analysis task instructions and local sentiment analysis task images.

[0124] Based on the same inventive concept, this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, it implements the steps in the instruction federation fine-tuning method for a multimodal large language model as described in the first aspect of this application.

[0125] Based on the same inventive concept, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the instruction federation fine-tuning method for a multimodal large language model as described in the first aspect of this application.

[0126] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0128] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0133] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0134] The foregoing has provided a detailed description of the instruction federation fine-tuning system, method, and product for a multimodal large language model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A federated fine-tuning system for a multimodal large language model, characterized in that, The first sentiment analysis task in the multiple sentiment analysis tasks is to classify the emotions in an image based on prompt text; the second sentiment analysis task is to describe the emotional cues in an image based on prompt text; the third sentiment analysis task is to locate objects with a target emotion category in an image based on prompt text; the first client in each client is used to fine-tune instructions for the first, second, and third sentiment analysis tasks, and the second client in each client is used to fine-tune instructions for the second and third sentiment analysis tasks; The third client among the various clients is used to fine-tune instructions for the first sentiment analysis task and the third sentiment analysis task; The system includes: The server sends global parameters for the current round of multiple sentiment analysis tasks to various clients, including: the server sends global parameters for the current round of the first sentiment analysis task to the first client and the third client, the server sends global parameters for the current round of the second sentiment analysis task to the first client and the second client, and the server sends global parameters for the current round of the third sentiment analysis task to the first client, the second client, and the third client; each client uses the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples to obtain a local task-conditional multimodal representation; Each client uses its local current-round personalized parameters to process its local task-conditional multimodal representation, thus obtaining its local task-conditional multimodal personalized representation. Each client fine-tunes its local multimodal large language model, which has global parameters and personalized parameters for the current task, based on its local task-conditional multimodal representation and local task-conditional multimodal personalized representation. The fine-tuned global parameters for the current task are uploaded to the server, while the fine-tuned personalized parameters are kept locally for use in the next round of instruction federated fine-tuning. The server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task based on the global parameters uploaded by each client, and obtains the global parameters of the next round of tasks for multiple sentiment analysis tasks. The server then sends the global parameters of the next round of tasks for multiple sentiment analysis tasks to each client. After the instruction federation fine-tuning is completed, each client obtains its local final personalized parameters, and the server obtains the final global parameters of multiple sentiment analysis tasks and sends them to each client. Each client utilizes a local multimodal large language model with local final personalized parameters and final task global parameters from the server to perform multiple sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images, including: After the instruction federation fine-tuning is completed, the first client obtains the final personalized parameters for the first sentiment analysis task, the second sentiment analysis task, and the third sentiment analysis task locally; the second client obtains the final personalized parameters for the second sentiment analysis task and the third sentiment analysis task locally; and the third client obtains the final personalized parameters for the first sentiment analysis task and the third sentiment analysis task locally. The server sends the final global parameters of the first sentiment analysis task to the first client and the third client, and sends the final global parameters of the second sentiment analysis task to the first client and the second client, and sends the final global parameters of the third sentiment analysis task to the first client, the second client and the third client. The first client utilizes a local multimodal large language model for the first sentiment analysis task, the second sentiment analysis task, and the third sentiment analysis task, and executes the first sentiment analysis task, the second sentiment analysis task, and the third sentiment analysis task based on local sentiment analysis task instructions and local sentiment analysis task images; The second client utilizes a local multimodal large language model for the second and third sentiment analysis tasks, and executes the second and third sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images. The third client utilizes a local multimodal large language model for the first and third sentiment analysis tasks, and executes the first and third sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images.

2. The instruction federation fine-tuning system for a multimodal large language model according to claim 1, characterized in that, Before the first round of instruction federation fine-tuning, the server sends the parameters of the frozen cross-modal alignment connector to each client. The frozen cross-modal alignment connector is used to map image features to image tokens. During each round of instruction federation fine-tuning, each client uses the parameters of the frozen cross-modal alignment connector to map the image features of local sentiment analysis task image samples into image tokens. During the process of fine-tuning the local multimodal large language model with global parameters and personalized parameters of the current task based on the local task-conditional multimodal representation and the local task-conditional multimodal personalized representation, each client keeps the parameters of the frozen cross-modal alignment connector unchanged for use in the next round of instruction federated fine-tuning. Each client utilizes the frozen cross-modal alignment connector, a local multimodal large language model with local final personalized parameters and final task global parameters from the server, to perform multiple sentiment analysis tasks for local sentiment analysis task instructions and local sentiment analysis task images.

3. The instruction federation fine-tuning system for a multimodal large language model according to claim 1, characterized in that, Each client has local parameters for a frozen cross-modal alignment connector used to map image features to image tokens; During each round of instruction federation fine-tuning, each client uses the parameters of the local frozen cross-modal alignment connector to map the image features of the local sentiment analysis task image samples into image tokens. During the fine-tuning of the local multimodal large language model with global parameters and personalized parameters of the current task based on the local task-conditional multimodal representation and the local task-conditional multimodal personalized representation, each client keeps the parameters of the local frozen cross-modal alignment connector unchanged for the next round of instruction federated fine-tuning. Each client utilizes a local frozen cross-modal alignment connector, a local multimodal large language model with local final personalized parameters and final task global parameters from the server, to perform multiple sentiment analysis tasks for local sentiment analysis task instructions and local sentiment analysis task images.

4. The instruction federation fine-tuning system for a multimodal large language model according to claim 1, characterized in that, Each client utilizes the semantics of its local sentiment analysis task instruction samples to modulate the image tokens of its local sentiment analysis task image samples, obtaining a local task-conditionalized multimodal representation, including: Each client obtains its local query vector based on the semantics of the local sentiment analysis task instruction samples; Each client obtains its own key vector and value vector based on the image tokens of the local sentiment analysis task image samples. Each client uses an attention mechanism to process its local query vector, key vector, and value vector to obtain a local task-conditional multimodal representation.

5. A multimodal large language model instruction federation fine-tuning system according to any one of claims 1-4, characterized in that, The plurality of sentiment analysis tasks include a first sentiment analysis task, and the multimodal training samples of the first client to the Kth client for the first sentiment analysis task are non-independent and identically distributed. For the first sentiment analysis task, each of the first client to the Kth client uploads the number of multimodal training samples for the first sentiment analysis task locally to the server; The server determines the total number of all multimodal training samples for the first sentiment analysis task, and determines the aggregation coefficient for each of the first to K clients based on the ratio of the number of samples uploaded by the first client to the Kth client to the total number. Based on the global parameters of the current round of the first sentiment analysis task uploaded by the first client to the Kth client after fine-tuning, the server performs a weighted summation of the global parameters of the current round of the first sentiment analysis task according to the aggregation coefficients of the first client to the Kth client, and then sends the global parameters of the next round of the first sentiment analysis task to the first client to the Kth client.

6. A multimodal large language model instruction federation fine-tuning system according to any one of claims 1-4, characterized in that, The first client among the various clients is used to execute M sentiment analysis tasks among the plurality of sentiment analysis tasks, and the first client is any one of the various clients; Take m as any integer from 1 to M. The first client reads the multimodal training samples for the m-th sentiment analysis task from the local machine according to the m-th sentiment analysis task identifier, so as to obtain the fine-tuned global parameters of the current round of tasks for the m-th sentiment analysis task and upload them to the server. The fine-tuned personalized parameters for the m-th sentiment analysis task are kept locally for the next round of instruction federated fine-tuning. After the instruction federation fine-tuning is completed, the first client obtains the final personalized parameters for the m-th sentiment analysis task locally, and the server obtains the final global parameters for the m-th sentiment analysis task and sends them to the first client. The first client utilizes a local multimodal large language model for the m-th sentiment analysis task, and executes the m-th sentiment analysis task based on the local sentiment analysis task instructions and the local sentiment analysis task images. The local multimodal large language model for the m-th sentiment analysis task has the final personalized parameters for the m-th sentiment analysis task and the final global parameters for the m-th sentiment analysis task from the server.

7. A multimodal large language model instruction federation fine-tuning system according to any one of claims 1-4, characterized in that, The server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task based on the fine-tuned global parameters uploaded by each client, and obtains the initial global parameters of the next round of tasks for multiple sentiment analysis tasks. The server uses multimodal training samples from multiple local sentiment analysis tasks to update the initial global parameters for the next round of the multiple sentiment analysis tasks, obtains the global parameters for the next round of the multiple sentiment analysis tasks, and sends them to each client.

8. A method for instruction federation fine-tuning of a multimodal large language model, characterized in that, The first sentiment analysis task in the multiple sentiment analysis tasks is to classify the emotions in an image based on prompt text; the second sentiment analysis task is to describe the emotional cues in an image based on prompt text; the third sentiment analysis task is to locate objects with a target emotion category in an image based on prompt text; the first client in each client is used to fine-tune instructions for the first, second, and third sentiment analysis tasks, and the second client in each client is used to fine-tune instructions for the second and third sentiment analysis tasks; The third client among the various clients is used to fine-tune instructions for the first sentiment analysis task and the third sentiment analysis task; The method includes: In response to the server issuing global parameters for multiple sentiment analysis tasks to various clients, the process includes: the server issuing global parameters for the first sentiment analysis task to the first client and the third client, issuing global parameters for the second sentiment analysis task to the first client and the second client, and issuing global parameters for the third sentiment analysis task to the first client, the second client, and the third client; each client modulates the image tokens of its local sentiment analysis task image samples using the semantics of its local sentiment analysis task instruction samples to obtain a local task-conditional multimodal representation. Each client uses its local current-round personalized parameters to process its local task-conditional multimodal representation, thus obtaining its local task-conditional multimodal personalized representation. Each client fine-tunes its local multimodal large language model, which has global parameters and personalized parameters for the current task, based on its local task-conditional multimodal representation and local task-conditional multimodal personalized representation. The fine-tuned global parameters for the current task are uploaded to the server, while the fine-tuned personalized parameters are kept locally for use in the next round of instruction federated fine-tuning. In response to the fine-tuned global parameters of the current round of tasks uploaded by each client, the server aggregates the fine-tuned global parameters of the current round of tasks for the same sentiment analysis task to obtain the global parameters of the next round of tasks for multiple sentiment analysis tasks, and sends the global parameters of the next round of tasks for multiple sentiment analysis tasks to each client. After the instruction federation fine-tuning is completed, each client obtains its local final personalized parameters, and the server obtains the final global parameters of multiple sentiment analysis tasks and sends them to each client. Each client utilizes a local multimodal large language model with local final personalized parameters and final task global parameters from the server to perform multiple sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images, including: After the instruction federation fine-tuning is completed, the first client obtains the final personalized parameters for the first sentiment analysis task, the second sentiment analysis task, and the third sentiment analysis task locally; the second client obtains the final personalized parameters for the second sentiment analysis task and the third sentiment analysis task locally; and the third client obtains the final personalized parameters for the first sentiment analysis task and the third sentiment analysis task locally. The server sends the final global parameters of the first sentiment analysis task to the first client and the third client, and sends the final global parameters of the second sentiment analysis task to the first client and the second client, and sends the final global parameters of the third sentiment analysis task to the first client, the second client and the third client. The first client utilizes a local multimodal large language model for the first sentiment analysis task, the second sentiment analysis task, and the third sentiment analysis task, and executes the first sentiment analysis task, the second sentiment analysis task, and the third sentiment analysis task based on local sentiment analysis task instructions and local sentiment analysis task images; The second client utilizes a local multimodal large language model for the second and third sentiment analysis tasks, and executes the second and third sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images. The third client utilizes a local multimodal large language model for the first and third sentiment analysis tasks, and executes the first and third sentiment analysis tasks based on local sentiment analysis task instructions and local sentiment analysis task images.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the steps in the instruction federation fine-tuning method for a multimodal large language model as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the instruction federation fine-tuning method for a multimodal large language model as described in claim 8.

Images