Federal learning-based pre-training model fine tuning method and related equipment

Abstract

The invention provides a federated learning-based pre-training model fine tuning method and related equipment, which are applied to first computing equipment corresponding to a server in federated learning, and a plurality of clients in federated learning hold private data corresponding to a target task to be finely tuned. The method comprises the following steps: sending a pre-training model and an initial global prompt parameter to a plurality of second computing devices corresponding to a plurality of clients, so that the plurality of second computing devices carry out prompt fine tuning on the pre-training model based on the initial global prompt parameter and held private data to obtain a private prompt parameter; obtaining a plurality of private prompt parameters obtained by the plurality of second computing devices, and aggregating the plurality of private prompt parameters to obtain an updated global prompt parameter; and based on the similarity between the updated global prompt parameter and each private prompt parameter, adjusting the updated global prompt parameter to obtain an adjusted global prompt parameter.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of pre-training model technology, and in particular, to a pre-training model fine-tuning method based on federated learning and related equipment. Background Art

[0002] A pre-trained model is a language model built with a deep neural network containing more than tens of billions of parameters. It uses self-supervised learning methods to master many language phenomena through pre-training on large and diverse public datasets. In order to apply the pre-trained model to a specific reasoning task, it is necessary to further fine-tune the pre-trained model using a dataset related to the reasoning task.

[0003] Fine-tuning the prompt of the pre-trained model based on federated learning is a popular fine-tuning method, which not only protects the data privacy of each participant in the prompt fine-tuning, but also can utilize a wide range of data from different sources to enhance the generalization ability of the pre-trained model. Specifically, multiple participants (i.e., clients) respectively use their private data and the initial global prompt parameters to fine-tune the pre-trained model locally, and send the fine-tuning training results (such as the prompt parameters after each fine-tuning) to the server for aggregation to obtain updated global prompt parameters. However, when summarizing the prompt parameters after fine-tuning by all parties, the server often directly calculates the average value of multiple prompt parameters, which can easily lead to the loss of the original Prompt information, greatly reducing the effect of prompt fine-tuning, and not meeting the actual needs of users. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a pre-training model fine-tuning method based on federated learning and related equipment.

[0005] In a first aspect, this specification provides a method for fine-tuning a pre-trained model based on federated learning, which is applied to a first computing device corresponding to a server in federated learning, wherein the server holds a pre-trained model to be fine-tuned; and multiple clients in federated learning hold private data corresponding to a target task to be fine-tuned; the method includes:

[0006] Sending the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to a plurality of second computing devices corresponding to the plurality of clients, so that the plurality of second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each of them, to obtain their own private prompt parameters;

[0007] Acquire multiple private prompt parameters obtained by the multiple second computing devices, and aggregate the multiple private prompt parameters to obtain updated global prompt parameters;

[0008] Calculate the similarity between the updated global prompt parameter and each private prompt parameter, and adjust the updated global prompt parameter based on the similarity to obtain an adjusted global prompt parameter; wherein the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

[0009] In a second aspect, the present specification provides a method for fine-tuning a pre-trained model based on federated learning, which is applied to any second computing device among multiple second computing devices corresponding to multiple clients in federated learning; the multiple clients hold private data corresponding to a target task to be fine-tuned; the method includes:

[0010] Obtaining a pre-trained model to be fine-tuned and an initial global prompt parameter for assisting the pre-trained model to perform the target task, which is sent by a first computing device corresponding to a server in federated learning;

[0011] Based on the initial global prompt parameters and the held private data, the pre-trained model is prompted and fine-tuned to obtain private prompt parameters, and the obtained private prompt parameters are sent to the first computing device, so that the first computing device obtains the multiple private prompt parameters obtained by the multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters, and calculates the similarity between the updated global prompt parameters and each private prompt parameter, and adjusts the updated global prompt parameters based on the similarity to obtain adjusted global prompt parameters; wherein the similarity between the adjusted global prompt parameters and each private prompt parameter is greater than the similarity between the updated global prompt parameters and each private prompt parameter.

[0012] In a third aspect, the present specification provides a pre-trained model fine-tuning device based on federated learning, which is applied to a first computing device corresponding to a server in federated learning, wherein the server holds a pre-trained model to be fine-tuned; multiple clients in federated learning hold private data corresponding to a target task to be fine-tuned; the device includes:

[0013] A first fine-tuning unit is used to send the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to a plurality of second computing devices corresponding to the plurality of clients, so that the plurality of second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each of them, to obtain the respective private prompt parameters;

[0014] a prompt parameter aggregation unit, configured to obtain a plurality of private prompt parameters obtained by the plurality of second computing devices, and aggregate the plurality of private prompt parameters to obtain updated global prompt parameters;

[0015] A prompt parameter adjustment unit is used to calculate the similarity between the updated global prompt parameter and each private prompt parameter, and adjust the updated global prompt parameter based on the similarity to obtain an adjusted global prompt parameter; wherein the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

[0016] In a fourth aspect, the present specification provides a pre-trained model fine-tuning device based on federated learning, which is applied to any second computing device among multiple second computing devices corresponding to multiple clients in federated learning; the multiple clients hold private data corresponding to a target task to be fine-tuned; the device includes:

[0017] An acquisition unit, used to acquire a pre-trained model to be fine-tuned sent by a first computing device corresponding to a server in federated learning and an initial global prompt parameter for assisting the pre-trained model to perform the target task;

[0018] A fine-tuning unit is used to perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the held private data to obtain private prompt parameters, and send the obtained private prompt parameters to the first computing device so that the first computing device obtains the multiple private prompt parameters obtained by the multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters, and calculates the similarity between the updated global prompt parameters and each private prompt parameter, and adjusts the updated global prompt parameters based on the similarity to obtain adjusted global prompt parameters; wherein the similarity between the adjusted global prompt parameters and each private prompt parameter is greater than the similarity between the updated global prompt parameters and each private prompt parameter.

[0019] Accordingly, the present specification also provides a computing device, comprising: a memory and a processor; the memory stores computer programs / instructions that can be executed by the processor; when the processor executes the computer program / instructions, it executes the pre-trained model fine-tuning method based on federated learning described in the first aspect or the second aspect above.

[0020] Accordingly, the present specification also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the pre-trained model fine-tuning method based on federated learning as described in the first aspect or the second aspect above is executed.

[0021] Accordingly, the present specification also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the pre-trained model fine-tuning method based on federated learning as described in the first aspect or the second aspect above is performed.

[0022] In summary, the present application can be applied to a first computing device corresponding to a server in federated learning, the server can hold a pre-trained model to be fine-tuned, and multiple clients in federated learning can respectively hold private data corresponding to the target task to be fine-tuned. In an illustrated embodiment, the first computing device can send the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to multiple second computing devices corresponding to multiple clients, so that the multiple second computing devices can respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each, and obtain their respective private prompt parameters. Further, the first computing device obtains multiple private prompt parameters obtained by multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters. Further, the first computing device calculates the similarity between the updated global prompt parameter and each private prompt parameter, and adjusts the updated global prompt parameter based on the similarity to obtain the adjusted global prompt parameter. Among them, the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter. In this way, after the application aggregates the multiple private prompt parameters obtained by the local prompt fine-tuning of multiple clients to obtain the updated global prompt parameters, the updated global prompt parameters are further adjusted to improve the similarity between the updated global prompt parameters and the multiple private prompt parameters, and the next round of prompt fine-tuning can be performed based on the adjusted global prompt parameters. As a result, when the server aggregates the private prompt parameters of multiple clients, it can avoid the loss of the original Prompt information contained in the private prompt parameters of multiple clients as much as possible, thereby improving the effect of prompt fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0024] Figure 2 It is a flowchart of a method for fine-tuning a pre-trained model based on federated learning provided by an exemplary embodiment;

[0025] Figure 3 It is a flowchart of another method for fine-tuning a pre-trained model based on federated learning provided by an exemplary embodiment;

[0026] Figure 4 It is a structural schematic diagram of a pre-training model fine-tuning device based on federated learning provided by an exemplary embodiment;

[0027] Figure 5 It is a structural schematic diagram of another pre-training model fine-tuning device based on federated learning provided by an exemplary embodiment;

[0028] Figure 6 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0030] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0031] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0032] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] First, some terms in this specification are explained to facilitate understanding by those skilled in the art.

[0034] (1) Large Language Models (LLMs), also known as pre-trained models, are language models built using deep neural networks with more than tens of billions of parameters. They use self-supervised learning methods to master many language phenomena through pre-training on large and diverse public datasets.

[0035] It should be understood that although the pre-trained model has demonstrated excellent capabilities in understanding and generating natural language, in order to apply the pre-trained model to reasoning tasks related to specific application scenarios, the pre-trained model needs to be further fine-tuned using data sets related to the specific reasoning task.

[0036] (2) Prompt Tuning, or prompt tuning, is a fine-tuning technique that guides pre-trained models to perform specific tasks, such as question-answering, image classification, sentiment analysis, etc., by designing text prompts. Prompts can be used as the context of the task or instructions, and are inserted before or during user input in order to allow the model to "understand" the current task requirements. For example, if the current task is sentiment analysis, Prompts may be a guiding sentence such as "What is the sentiment of this comment?" For another example, if the current task is image classification, Prompt may be a guiding sentence such as "What is described in the picture?" This manual does not make specific restrictions on this.

[0037] Unlike Fine-Tuning, which involves fine-tuning all parameters of the entire pre-trained model to adapt to a specific task, Fine-Tuning is effective, but it may require a lot of computing resources and time when facing large models, and there is a high risk of overfitting. Prompt Tuning only adds a number of small-scale trainable parameter sets, namely the embedding vectors corresponding to the prompt, to the fixed pre-trained model. This means that Prompt Tuning only needs to optimize a small number of parameters, which greatly reduces the computational burden while retaining the generalization ability of the pre-trained model.

[0038] (3) Federated Learning (FL) is a distributed machine learning framework that allows various participants (usually referred to as clients) to collaboratively train a shared global model without centralizing the private data held by each client in one place. In other words, each client does not need to share its own private data. Instead, each client only needs to aggregate its local training results (such as updated model parameters or gradient information) to the server. This allows learning from multiple data sources and improving the model's capabilities while protecting data privacy and security.

[0039] Based on this, a federated learning method can be used to combine multiple clients to perform prompt fine-tuning on the pre-trained model. Specifically, the server in the federated learning can first design an initial global prompt parameter based on the current target task to be fine-tuned, and distribute the initial global prompt parameter to multiple clients. Furthermore, each client can use its private data and the initial global prompt parameter to fine-tune the pre-trained model locally, and then upload the prompt fine-tuning results (such as the prompt parameters after each fine-tuning) to the central server for aggregation, thereby obtaining updated global prompt parameters. Subsequently, the central server distributes the updated global prompt parameters to each client, so that each client further fine-tunes the pre-trained model based on the updated global prompt parameter, and so on, and repeatedly performs the above process until the predetermined fine-tuning target is reached.

[0040] As mentioned above, the prompt fine-tuning based on federated learning not only protects the data privacy of each participant, but also can utilize a wide range of data from different sources to enhance the generalization ability of the pre-trained model. However, when the server aggregates the prompt parameters fine-tuned by each party, it often directly calculates the average of multiple prompt parameters. For example, client C1 and client C2 perform prompt fine-tuning locally, and the fine-tuned prompts are: Prompt#1 = "A photo of a[CLASS]", Prompt#2 = "A photo describing a[CLASS]". The average prompt obtained by the server directly averaging the tokens in the embedding space is: Prompt#Average = "A photo[X]a[CLASS]", where [X] is a word without any semantic meaning obtained by averaging the embedding vectors of "of" and "describing", which leads to the loss of the original prompt information, greatly reduces the effect of prompt fine-tuning, and does not meet the actual needs of users.

[0041] Based on this, this specification provides a technical solution, which, after simply aggregating the prompt parameters of multiple clients to obtain updated global prompt parameters, further adjusts the updated global prompt parameters to improve the similarity between the global prompt parameters and the prompt parameters of multiple clients, thereby avoiding the loss of the original Prompt information contained in the prompt parameters of multiple clients as much as possible and improving the effect of prompt fine-tuning.

[0042] In implementation, the technical solution can be applied to a first computing device corresponding to a server in federated learning, the server can hold a pre-trained model to be fine-tuned, and multiple clients in federated learning can respectively hold private data corresponding to the target task to be fine-tuned. In an illustrated embodiment, the first computing device can send the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to multiple second computing devices corresponding to multiple clients, so that the multiple second computing devices can respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each, and obtain their respective private prompt parameters. Further, the first computing device obtains multiple private prompt parameters obtained by multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters. Further, the first computing device calculates the similarity between the updated global prompt parameter and each private prompt parameter, and adjusts the updated global prompt parameter based on the similarity to obtain the adjusted global prompt parameter. Among them, the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

[0043] In the above technical solution, after aggregating multiple private prompt parameters obtained by fine-tuning prompts locally on multiple clients to obtain updated global prompt parameters, the application can further adjust the updated global prompt parameters to improve the similarity between the updated global prompt parameters and multiple private prompt parameters, and then perform the next round of prompt fine-tuning based on the adjusted global prompt parameters. In this way, when the server aggregates the private prompt parameters of multiple clients, it can avoid the loss of the original Prompt information contained in the private prompt parameters of multiple clients as much as possible, thereby improving the effect of prompt fine-tuning.

[0044] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a system architecture provided by an exemplary embodiment. Figure 1 As shown, the system may include a first computing device 100 (i.e., a central server) corresponding to a server in federated learning, and multiple second computing devices corresponding to multiple clients in federated learning, such as a second computing device 200a, a second computing device 200b, and a second computing device 200c. In an illustrated embodiment, the first computing device 100, and the second computing devices 200a, 200b, and 200c may establish a communication connection in any possible manner, and this specification does not specifically limit this.

[0045] Among them, the server holds a pre-trained model that is publicly available and expects that the pre-trained model can successfully perform the target tasks related to the target application scenario through further fine-tuning training.

[0046] Among them, multiple clients hold private data corresponding to the target task. Exemplarily, the above-mentioned target task can be, for example, asset analysis, traffic planning, user interface development, image generation, image classification, image visual description or question answering tasks, etc., and this specification does not make specific limitations on this. Exemplarily, the above-mentioned private data can include, for example, math data corresponding to the asset analysis task, code data corresponding to the user interface development task, and image-text data sets corresponding to image generation, image visual description and image classification tasks, etc., and this specification does not make specific limitations on this.

[0047] In an illustrated embodiment, the first computing device 100 corresponding to the server can send the pre-trained model to be fine-tuned to the second computing device 200a, the second computing device 200b, and the second computing device 200c corresponding to the plurality of clients. Further, the first computing device 100 can design an initial global prompt parameter according to the above target task, or can randomly initialize a global prompt parameter, and send the initial global prompt parameter to the second computing device 200a, the second computing device 200b, and the second computing device 200c.

[0048] Among them, the global prompt parameter can be used to assist the pre-trained model to perform the target task. Specifically, the global prompt parameter may include: multiple embedding vectors for describing the semantic information of multiple words contained in the prompt text corresponding to the global prompt parameter; wherein, the prompt text can be input into the pre-trained model to assist the pre-trained model to perform the target task.

[0049] Accordingly, the second computing device 200a, the second computing device 200b, and the second computing device 200c may receive the pre-trained model to be fine-tuned and the initial global prompt parameters sent by the first computing device 100. Further, the second computing device 200a, the second computing device 200b, and the second computing device 200c may respectively perform prompt fine-tuning on the pre-trained model locally based on the initial global prompt parameters and the private data held by each, and obtain their own private prompt parameters.

[0050] It should be understood that the purpose of prompt tuning is mainly to continuously optimize the prompt parameters to better assist the pre-trained model to understand and execute the target task, thereby outputting more accurate reasoning results. Accordingly, the private prompt parameters obtained by each second computing device are based on local private data to perform prompt fine-tuning, so that the prompt parameters obtained after adjusting and optimizing the initial global prompt parameters can better assist the pre-trained model to execute the target task. Specifically, the private prompt parameters may include: multiple embedded vectors for describing the semantic information of multiple words contained in the prompt text corresponding to the private prompt parameters; wherein, the prompt text can be input into the pre-trained model to assist the pre-trained model to execute the target task.

[0051] For example, the prompt text corresponding to the private prompt parameter obtained by the second computer device 200a after local prompt fine-tuning is, for example, "A photo of a[CLASS]". The private prompt parameter may include the embedding vectors (or word vectors) of the five words "A", "photo", "of", "a", and "[CLASS]" in the embedding space.

[0052] Exemplarily, the prompt text corresponding to the private prompt parameter obtained by the second computer device 200b after fine-tuning the prompt locally is, for example, "A photo describing a[CLASS]", then the private prompt parameter may include the embedding vectors of the five words "A", "photo", "describing", "a", and "[CLASS]" in the embedding space.

[0053] Further, the second computing device 200a, the second computing device 200b, and the second computing device 200c can send the private prompt parameters obtained by their respective fine-tuning to the first computing device 100. Accordingly, the first computing device 100 obtains the private prompt parameters obtained by the second computing device 200a, the second computing device 200b, and the second computing device 200c, and aggregates these multiple private prompt parameters to obtain updated global prompt parameters. For details, please refer to the following Figure 2 The description of the corresponding embodiments will not be elaborated in detail here.

[0054] Furthermore, the first computing device 100 may also calculate the similarity between the updated global prompt parameter and each private prompt parameter, and adjust the updated global prompt parameter based on the similarity to obtain the adjusted global prompt parameter. For details, refer to the following Figure 2 The description of the corresponding embodiment is not further elaborated here. The similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

[0055] Furthermore, the first computing device 100 may send the adjusted global prompt parameters to the second computing device 200a, the second computing device 200b, and the second computing device 200c, so that the second computing device 200a, the second computing device 200b, and the second computing device 200c perform the next round of prompt fine-tuning based on the adjusted global prompt parameters, and so on, until the fine-tuning target corresponding to the above-mentioned target task is met.

[0056] In an illustrated embodiment, the present application may specifically adopt horizontal federated learning when fine-tuning the pre-trained model based on federated learning. Horizontal federated learning is a form of federated learning, which is usually applicable to situations where the data feature space is similar but the samples are not exactly the same, that is, multiple clients hold private data related to the target task, which is a data set containing the same features but different individual data.

[0057] In one illustrated embodiment, the first computing device 100, and the second computing device 200a, the second computing device 200b, and the second computing device 200c may be a smart wearable device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a server, or a server cluster or a cloud computing center composed of multiple servers, etc., which are not specifically limited in this specification.

[0058] See also Figure 2 , Figure 2 This is a flowchart of a method for fine-tuning a pre-trained model based on federated learning provided by an exemplary embodiment. This method can be applied to Figure 1 The system architecture shown can be applied to Figure 1 The first computing device corresponding to the server in federated learning is shown, wherein the server holds a pre-trained model to be fine-tuned, and multiple clients in federated learning hold private data corresponding to the target task to be fine-tuned. Figure 2 As shown, the method may specifically include the following steps S201-S202.

[0059] Step S201, sends the pre-trained model to be fine-tuned and the initial global prompt parameters used to assist the pre-trained model in performing the target task to multiple second computing devices corresponding to multiple clients, so that the multiple second computing devices can perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data they hold, and obtain their own private prompt parameters.

[0060] As described above, the present application can adopt the architecture of federated learning to perform prompt fine-tuning on the pre-trained model, wherein the server in the federated learning holds the pre-trained model to be fine-tuned, and multiple clients in the federated learning hold private data corresponding to the target task to be fine-tuned.

[0061] First, in an illustrated embodiment, a first computing device corresponding to the server may transmit a pre-trained model to be fine-tuned and an initial global prompt parameter (which may be denoted as P) for assisting the pre-trained model to perform a target task. g ) is sent to multiple second computing devices corresponding to multiple clients. In an illustrated embodiment, the explanation of the global prompt parameter can refer to the above Figure 1 The description of the corresponding embodiments will not be repeated here.

[0062] Accordingly, the plurality of second computing devices may receive the pre-trained model to be fine-tuned and the initial global prompt parameters sent by the first computing device. Further, the plurality of second computing devices may locally perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each of them, and obtain their own private prompt parameters. In an illustrated embodiment, the explanation of the private prompt parameters can be referred to the above Figure 1 The description of the corresponding embodiments will not be repeated here.

[0063] It should be noted that this specification does not specifically limit the specific implementation method of prompt fine-tuning. In one embodiment shown, the above-mentioned pre-trained model can be a pre-trained visual language (Contrastive Language-Image Pre-training, CLIP) model, and accordingly, multiple second computing devices can use the contextualized optimization of prompts (Contextualized Optimization of Prompts, CoOP) method to locally fine-tune the CLIP model, iterate E epochs, to complete the local fine-tuning, so as to obtain their own private prompt parameters, etc. This specification does not specifically limit this.

[0064] Epochs is an important hyperparameter in the neural network training process, which is defined as a single training iteration of all batches in forward and backward propagation. In short, an epoch is to input data into the network (i.e. the pre-trained model to be fine-tuned) to complete a forward calculation and backward propagation. It will not be described in detail here. E is an integer greater than or equal to 1.

[0065] Among them, the CLIP model is a multimodal pre-trained model that can understand the relationship between images and related description text. It learns a space by performing contrastive learning on a large number of image and text pairs on web pages. In this space, similar images and text representations are similar. Based on this, the CLIP model can be used to perform a variety of visual tasks, such as image classification, object detection, etc. Among them, CoOP is a prompt optimization technology, which is mainly used to improve the performance of the CLIP model on specific tasks. Different from the traditional PromptTuning, CoOp proposes to use learnable prompts in the visual field. During the training process, CoOp improves the performance of the model in visual tasks by optimizing the text part in the prompt, without retraining or fine-tuning the entire model, etc., which will not be elaborated here.

[0066] Step S202: Acquire multiple private prompt parameters obtained by the multiple second computing devices, and aggregate the multiple private prompt parameters to obtain updated global prompt parameters.

[0067] Furthermore, after the multiple second computing devices complete the prompt fine-tuning locally and obtain their own private prompt parameters, they can transmit the private prompt parameters back to the first computing device. Accordingly, the first computing device obtains the multiple private prompt parameters obtained by the fine-tuning of the multiple second computing devices.

[0068] In an illustrated embodiment, a set of multiple private prompt parameters may be denoted as {P 1 , P 2 ,…,P N}, where the i-th private hint parameter P in the set i That is, the private prompt parameter obtained by the i-th second computing device after local fine-tuning. Wherein, N is an integer greater than 1, and i is an integer greater than or equal to 1 and less than or equal to N.

[0069] Furthermore, the first computing device may aggregate the above-mentioned multiple private prompt parameters to obtain updated global prompt parameters (which may be recorded as

[0070] It should be noted that this specification does not specifically limit the specific implementation method of aggregating multiple private prompt parameters.

[0071] In an illustrated embodiment, the first computing device may simply use the average value of the multiple private prompt parameters as the updated global prompt parameter

[0072] In an illustrated embodiment, the first computing device may first determine the weights of the plurality of private prompt parameters, and then perform weighted average calculation on the plurality of private prompt parameters according to the determined weights, and use the weighted average calculation result as the updated global prompt parameter. In one illustrated embodiment, the first computing device can determine the respective weights of multiple private hint parameters based on the model output effects corresponding to the multiple private hint parameters (i.e., the accuracy of the inference results output by the pre-trained model in the local fine-tuning stage), etc., and this specification does not make any specific limitations on this.

[0073] Specifically, the first computing device performs an average calculation or a weighted average calculation on multiple embedding vectors included in multiple private prompt parameters in the embedding space, thereby obtaining an updated global prompt parameter

[0074] For example, still taking the above “A photo of a[CLASS]” and “A photo describing a[CLASS]” as examples, the average prompt obtained by the first computing device in the embedding space by averaging the tokens is: Prompt#Average = “A photo[X]a[CLASS]”, where [X] is a word without any semantic meaning obtained by averaging the embedding vectors of “of” and “describing”. Accordingly, the updated global prompt parameter It can include the embedding vectors of the five words "A", "photo", "[X]", "a", and "[CLASS]" in the embedding space.

[0075] Step S203, calculating the similarity between the updated global prompt parameter and each private prompt parameter, and adjusting the updated global prompt parameter based on the similarity to obtain the adjusted global prompt parameter; wherein the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

[0076] Further, the first computing device aggregates the private prompt parameter set {P 1 , P 2 ,…,P N}, get the updated global prompt parameters After that, the updated global prompt parameter can be calculated With the private hint parameter P 1 , P 2 ,…,P N The similarity between each private hint parameter in , and based on the similarity, the updated global hint parameter Make adjustments to obtain the adjusted global prompt parameters.

[0077] Exemplarily, the first computing device calculates the updated global prompt parameter With the private hint parameter P 1 , P 2 ,…,P N When the similarity between each private hint parameter in is calculated, a total of N similarities can be obtained. Further, the average value of the N similarities can be calculated, and then the updated global hint parameter is updated based on the average value of the N similarities. Make adjustments to obtain adjusted global prompt parameters.

[0078] Among them, the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the updated global prompt parameter The similarity between the adjusted global prompt parameter and the private prompt parameter P 1 , P 2 ,…,P N The average of the N similarities between is greater than the updated global hint parameter With the private hint parameter P 1 , P 2 ,…,P N The average of the N similarities between them.

[0079] Take N = 3 as an example, assuming the updated global prompt parameter is With the private hint parameter P 1 , P 2 , P 3 The three similarities between them are 0.3, 0.4, and 0.2 respectively, then the average of the three similarities is equal to 0.3; and the adjusted global prompt parameter and private prompt parameter P 1 , P 2 , P 3 The three similarities between them are 0.7, 0.8, and 0.6 respectively, then the average of the three similarities is equal to 0.7, which is greater than the updated global prompt parameter With the private hint parameter P 1 , P 2 , P 3 The average of the three similarities is 0.3.

[0080] It should be noted that this specification does not refer to the global prompt parameters updated based on similarity. The specific implementation method of the adjustment is not particularly limited. In an embodiment shown, the updated global prompt parameter Perform multiple rounds of adjustments to obtain the final adjusted global prompt parameters. Specifically, the first computing device first calculates the updated global prompt parameters The similarity between each private hint parameter and the updated global hint parameter based on the similarity Adjustment is performed, and further, the similarity between the adjusted current global prompt parameter and each private prompt parameter is calculated, and the global prompt parameter is further adjusted based on the similarity calculated again, and so on, until the adjustment target is met, and the adjusted global prompt parameter is obtained.

[0081] In an illustrated embodiment, the above-mentioned adjustment target may include: the number of adjustments reaches a preset number threshold, or the similarity between the adjusted global prompt parameter and each private prompt parameter is less than a preset similarity threshold, or the average value of the similarity between the adjusted global prompt parameter and multiple private prompt parameters is less than a preset similarity threshold, etc. This specification does not make specific limitations on this.

[0082] In one illustrated implementation, the pre-trained model may be a CLIP model, and the CLIP model includes a text encoder (TextEncoder).

[0083] Based on this, the first computing device calculates the updated global prompt parameter The similarity between each private prompt parameter may include: Input into the text encoder of the CLIP model to obtain the updated global prompt parameters The corresponding global hidden layer representation; wherein the global hidden layer representation may include: a hidden layer space vector for describing the semantic information contained in the prompt text corresponding to the updated global prompt parameter. And each private prompt parameter is input into the text encoder to obtain a private hidden layer representation corresponding to each private prompt parameter; wherein the private hidden layer representation may include: a hidden layer space vector for describing the semantic information contained in the prompt text corresponding to its private prompt parameter. Then, the vector similarity between the global hidden layer representation and each private hidden layer representation is calculated, i.e., the updated global prompt parameter The similarity with each private hint parameter may include a vector similarity between the corresponding global hidden layer representation and each private hidden layer representation.

[0084] For example, the updated global prompt parameter Taking the embedding vectors of the five words "A", "photo", "[X]", "a", and "[CLASS]" as examples, the global hidden layer representation obtained after inputting these three embedding vectors into the text encoder is a hidden space vector used to describe the semantic information of the complete sentence "A photo[X]a[CLASS]".

[0085] For example, any private prompt parameter P i Taking the embedding vectors of the five words "A", "photo", "of", "a", and "[CLASS]" as an example, the private hidden layer representation E obtained by inputting these three embedding vectors into the text encoder is i is a latent space vector used to describe the semantic information of the complete sentence “A photo of a[CLASS]”.

[0086] For example, any private prompt parameter P i Taking the embedding vectors of the five words "A", "photo", "describing", "a", and "[CLASS]" as examples, the private hidden layer representation E obtained by inputting these three embedding vectors into the text encoder is i is a latent space vector used to describe the semantic information of the complete sentence “A photo describing a[CLASS]”.

[0087] In some possible implementations, the first computing device may also calculate the global hidden layer representation and multiple private hidden layer representations in the hidden layer space in any other possible manner except the above-mentioned text encoder, and this specification does not make any specific limitation on this.

[0088] In an illustrated embodiment, the set of multiple private hidden layer representations corresponding to multiple private hint parameters can be recorded as {E 1 , E 2 , …, E N}, where the i-th hidden layer in the set represents E i That is the i-th private prompt parameter P i The corresponding private hidden layer representation.

[0089] As described above, the first computing device obtains the above-mentioned global hidden layer representation and the private hidden layer representation set {E 1 , E 2 , …, E N}, the global hidden layer representation and the private hidden layer representation E can be calculated 1 , E 2 , …, E N The vector similarity between each private hidden layer representation in .

[0090] Exemplarily, the first computing device calculates the global hidden layer representation and the private hidden layer representation E 1 , E 2 , …, E N When the vector similarity between each private hidden layer representation in , a total of N vector similarities can be obtained, and further, the average value of the N vector similarities can be calculated. Further, the global hint parameter for updating can be calculated based on the average value of the N vector similarities. The adjustment amount is adjusted, and the global prompt parameters are updated based on the adjustment amount. Make adjustments.

[0091] In an embodiment shown, the average value of the similarities of the N vectors relative to the initial global prompt parameter P can be calculated. gThe gradient information of the updated global prompt parameter is then calculated based on the gradient information, and the adjustment amount for adjusting the updated global prompt parameter is calculated, and so on. This specification does not make specific limitations on this.

[0092] In an illustrated implementation, the vector similarity may be cosine similarity, or any other possible method may be used to calculate the vector similarity, which is not specifically limited in this specification.

[0093] In an illustrated embodiment, the first computing device may specifically calculate the similarity using the following formula (1), and update the global prompt parameter Perform multiple rounds of adjustments:

[0094]

[0095] in, is the current global prompt parameter, η is The learning rate, The calculation is the i-th private hint parameter P i The corresponding private hidden layer representation E i With the current global prompt parameters The corresponding global hidden layer representation The cosine similarity between The global hidden layer representation is calculated With the private hidden layer representation E 1 , E 2 , …, E N The average of the N cosine similarities between Indicates the average value relative to the initial global prompt parameter P g The gradient information.

[0096] After the first computing device performs R rounds of iterations shown in the above formula (1), the final adjusted global prompt parameter can be obtained: That is, τ in formula (1) takes values ​​from 0 to R-1 in sequence, where R is an integer greater than 1. The specific value can be set according to actual conditions and requirements, and this specification does not make any specific limitations on this.

[0097] Further, the first computing device may adjust the global prompt parameters (eg Send to multiple second computing devices. Accordingly, multiple second computing devices can receive the adjusted global prompt parameters, and further perform prompt fine-tuning on the pre-trained model based on the adjusted global prompt parameters and the private data held by each, and so on, perform multiple rounds of prompt fine-tuning until the output of the pre-trained model meets the fine-tuning target corresponding to the target task to be fine-tuned, and so on, which is not specifically limited in this specification.

[0098] See also Figure 3 , Figure 3 FIG. 1 is a flowchart of another method for fine-tuning a pre-trained model based on federated learning provided by an exemplary embodiment. The method can be applied to Figure 1 The system architecture shown can be applied to Figure 1 Any second computing device among the plurality of second computing devices corresponding to the plurality of clients in the federated learning, wherein the plurality of clients hold private data corresponding to the target task to be fine-tuned. Figure 3 As shown, the method may specifically include the following steps S301-S302.

[0099] Step S301, obtaining a pre-trained model to be fine-tuned and initial global prompt parameters for assisting the pre-trained model to perform a target task, which are sent by a first computing device corresponding to a server in federated learning.

[0100] In an embodiment shown, step S301 may be specifically referred to above. Figure 2 The description in the corresponding embodiment will not be repeated here.

[0101] Step S302: fine-tune the pre-trained model based on the initial global prompt parameters and the private data held to obtain private prompt parameters, and send the obtained private prompt parameters to the first computing device.

[0102] In an embodiment shown, step S302 may be specifically referred to above. Figure 2 The description in the corresponding embodiment will not be repeated here.

[0103] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a pre-training model fine-tuning device based on federated learning. Figure 4 , Figure 4 40 is a schematic diagram of a pre-training model fine-tuning device based on federated learning provided by an exemplary embodiment. Figure 1 The system architecture shown can be applied to Figure 1 The first computing device shown corresponds to the server in federated learning, wherein the server holds a pre-trained model to be fine-tuned; and multiple clients in federated learning hold private data corresponding to the target task to be fine-tuned. Figure 4 As shown, the device 40 includes:

[0104] The first fine-tuning unit 401 is used to send the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to multiple second computing devices corresponding to the multiple clients, so that the multiple second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each, so as to obtain the respective private prompt parameters;

[0105] A prompt parameter aggregation unit 402, configured to obtain a plurality of private prompt parameters obtained by the plurality of second computing devices, and aggregate the plurality of private prompt parameters to obtain updated global prompt parameters;

[0106] The prompt parameter adjustment unit 403 is used to calculate the similarity between the updated global prompt parameter and each private prompt parameter, and adjust the updated global prompt parameter based on the similarity to obtain the adjusted global prompt parameter; wherein the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

[0107] In an illustrated embodiment, the device 40 further includes a second fine-tuning unit 404, which is used to:

[0108] The adjusted global prompt parameters are sent to the multiple second computing devices, so that the multiple second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the adjusted global prompt parameters and the private data held by each of them, and so on, until the output of the pre-trained model meets the fine-tuning target corresponding to the target task.

[0109] In an illustrated embodiment, the global prompt parameters include: multiple embedding vectors for describing the semantic information of multiple words contained in the prompt text corresponding to the global prompt parameters; the private prompt parameters include: multiple embedding vectors for describing the semantic information of multiple words contained in the prompt text corresponding to the private prompt parameters; wherein the prompt text is used to be input into the pre-trained model to assist the pre-trained model in performing the target task.

[0110] In an illustrated embodiment, the prompt parameter aggregation unit 402 is specifically configured to:

[0111] A weighted average calculation is performed on the multiple private prompt parameters, and the weighted average calculation result is used as the updated global prompt parameter.

[0112] In an illustrated embodiment, the prompt parameter adjustment unit 403 is specifically configured to:

[0113] Calculating a similarity between the updated global prompt parameter and each private prompt parameter, and adjusting the updated global prompt parameter based on the similarity;

[0114] The similarity between the current global prompt parameter and each private prompt parameter is calculated, and the global prompt parameter is further adjusted based on the similarity, and so on, until the adjustment target is met, thereby obtaining the adjusted global prompt parameter.

[0115] In an illustrated embodiment, the adjustment target includes: the number of adjustments reaches a preset number threshold, or the similarity between the adjusted global prompt parameter and each private prompt parameter is less than a preset similarity threshold.

[0116] In an illustrated embodiment, the pre-trained model includes a pre-trained visual language CLIP model, and the CLIP model includes a text encoder; the prompt parameter adjustment unit 403 is specifically used to:

[0117] Inputting the updated global prompt parameter into the text encoder to obtain a global hidden layer representation corresponding to the updated global prompt parameter; the global hidden layer representation includes: a hidden layer space vector for describing the semantic information contained in the prompt text corresponding to the updated global prompt parameter; and,

[0118] Input each private prompt parameter into the text encoder to obtain a private hidden layer representation corresponding to each private prompt parameter; the private hidden layer representation includes: a hidden layer space vector for describing the semantic information contained in the prompt text corresponding to the private prompt parameter;

[0119] A vector similarity between the global hidden layer representation and each private hidden layer representation is calculated.

[0120] In one illustrated embodiment, the vector similarity comprises cosine similarity.

[0121] In an illustrated embodiment, the prompt fine-tuning method adopted by the plurality of second computing devices when respectively performing prompt fine-tuning on the pre-trained model includes a prompt contextualization optimization method CoOP.

[0122] Accordingly, the embodiments of this specification also provide another pre-training model fine-tuning device based on federated learning. Figure 5 , Figure 5 FIG. 5 is a schematic diagram of another structure of a pre-training model fine-tuning device based on federated learning provided by an exemplary embodiment. The device 50 can be applied to Figure 1 The system architecture shown can be applied to Figure 1Any second computing device among the plurality of second computing devices corresponding to the plurality of clients in the federated learning, wherein the plurality of clients hold private data corresponding to the target task to be fine-tuned. Figure 5 As shown, the device 50 comprises:

[0123] An acquisition unit 501 is used to acquire a pre-trained model to be fine-tuned sent by a first computing device corresponding to a server in federated learning and an initial global prompt parameter for assisting the pre-trained model to perform the target task;

[0124] The fine-tuning unit 502 is used to perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the held private data to obtain private prompt parameters, and send the obtained private prompt parameters to the first computing device, so that the first computing device obtains the multiple private prompt parameters obtained by the multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters, and calculates the similarity between the updated global prompt parameters and each private prompt parameter, and adjusts the updated global prompt parameters based on the similarity to obtain adjusted global prompt parameters; wherein the similarity between the adjusted global prompt parameters and each private prompt parameter is greater than the similarity between the updated global prompt parameters and each private prompt parameter.

[0125] The implementation process of the functions and effects of each unit in the above-mentioned device 40 and device 50 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 40 and device 50 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0127] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0128] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Figure 6 , Figure 6 is a schematic diagram of a computing device provided by an exemplary embodiment. The computing device may be Figure 1 The first computing device corresponding to the server in the federated learning is shown, or any second computing device among multiple second computing devices corresponding to multiple clients in the federated learning. The server holds the pre-trained model to be fine-tuned; and multiple clients hold private data corresponding to the target task to be fine-tuned. Figure 6 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 6 As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be executed by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the pre-training model fine-tuning method based on federated learning in the embodiment of this specification.

[0129] For a detailed description of the various steps of the above-mentioned pre-trained model fine-tuning method based on federated learning, please refer to the previous content and will not be repeated here.

[0130] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when these computer programs are executed by the processor, the various steps of the pre-training model fine-tuning method based on federated learning in the embodiment of this specification are executed. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0131] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0132] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0133] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0134] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0135] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A pre-training model fine-tuning method based on federated learning, characterized in that: A first computing device corresponding to a server in federated learning, wherein the server holds a pre-trained model to be fine-tuned; Multiple clients in federated learning hold private data corresponding to a target task to be fine-tuned; the method includes: Sending the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to a plurality of second computing devices corresponding to the plurality of clients, so that the plurality of second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each of them, to obtain their own private prompt parameters; Acquire multiple private prompt parameters obtained by the multiple second computing devices, and aggregate the multiple private prompt parameters to obtain updated global prompt parameters; Calculate the similarity between the updated global prompt parameter and each private prompt parameter, and adjust the updated global prompt parameter based on the similarity; wherein the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

2. The method according to claim 1, characterized in that The method further comprises: The adjusted global prompt parameters are sent to the multiple second computing devices, so that the multiple second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the adjusted global prompt parameters and the private data held by each of them, and so on, until the output of the pre-trained model meets the fine-tuning target corresponding to the target task.

3. The method according to claim 1, characterized in that The global prompt parameters include: multiple embedding vectors for describing the semantic information of multiple words contained in the prompt text corresponding to the global prompt parameters; the private prompt parameters include: multiple embedding vectors for describing the semantic information of multiple words contained in the prompt text corresponding to the private prompt parameters; wherein the prompt text is used to be input into the pre-trained model to assist the pre-trained model in performing the target task.

4. The method according to claim 1, characterized in that: The aggregating the multiple private prompt parameters to obtain updated global prompt parameters includes: A weighted average calculation is performed on the multiple private prompt parameters, and the weighted average calculation result is used as the updated global prompt parameter.

5. The method according to claim 1, characterized in that The calculating the similarity between the updated global prompt parameter and each private prompt parameter, and adjusting the updated global prompt parameter based on the similarity, comprises: Calculating a similarity between the updated global prompt parameter and each private prompt parameter, and adjusting the updated global prompt parameter based on the similarity; The similarity between the current global prompt parameter and each private prompt parameter is calculated, and the global prompt parameter is further adjusted based on the similarity, and so on, until the adjustment target is met, thereby obtaining the adjusted global prompt parameter.

6. The method according to claim 5, characterized in that The adjustment target includes: the number of adjustments reaches a preset number threshold, or the similarity between the adjusted global prompt parameter and each private prompt parameter is less than a preset similarity threshold.

7. The method according to any one of claims 1 to 6, characterized in that: The pre-trained model includes a pre-trained visual language CLIP model, and the CLIP model includes a text encoder; and the calculating the similarity between the updated global prompt parameter and each private prompt parameter includes: Inputting the updated global prompt parameter into the text encoder to obtain a global hidden layer representation corresponding to the updated global prompt parameter; the global hidden layer representation includes: a hidden layer space vector for describing the semantic information contained in the prompt text corresponding to the updated global prompt parameter; and, Input each private prompt parameter into the text encoder to obtain a private hidden layer representation corresponding to each private prompt parameter; the private hidden layer representation includes: a hidden layer space vector for describing the semantic information contained in the prompt text corresponding to the private prompt parameter; A vector similarity between the global hidden layer representation and each private hidden layer representation is calculated.

8. The method according to claim 7, characterized in that The adjusting the updated global prompt parameter based on the similarity includes: Calculating an average value of vector similarities between the global hidden layer representation and each private hidden layer representation, and calculating gradient information of the average value relative to the initial global hint parameter; Based on the gradient information, an adjustment amount for adjusting the updated global prompt parameter is calculated, and the updated global prompt parameter is adjusted based on the adjustment amount.

9. The method according to claim 7, characterized in that: The vector similarity includes cosine similarity.

10. The method according to claim 7, characterized in that The prompt fine-tuning methods adopted by the multiple second computing devices when respectively fine-tuning the pre-trained models include a prompt contextualization optimization method CoOP.

11. A method for fine-tuning a pre-trained model based on federated learning, characterized in that: Applied to any second computing device among a plurality of second computing devices corresponding to a plurality of clients in the federated learning; The plurality of clients hold private data corresponding to the target task to be fine-tuned; the method comprises: Obtaining a pre-trained model to be fine-tuned and an initial global prompt parameter for assisting the pre-trained model to perform the target task, which is sent by a first computing device corresponding to a server in federated learning; Based on the initial global prompt parameters and the held private data, the pre-trained model is prompted and fine-tuned to obtain private prompt parameters, and the obtained private prompt parameters are sent to the first computing device, so that the first computing device obtains the multiple private prompt parameters obtained by the multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters, and calculates the similarity between the updated global prompt parameters and each private prompt parameter, and adjusts the updated global prompt parameters based on the similarity; wherein the similarity between the adjusted global prompt parameters and each private prompt parameter is greater than the similarity between the updated global prompt parameters and each private prompt parameter.

12. A pre-training model fine-tuning device based on federated learning, characterized in that: A first computing device corresponding to a server in federated learning, wherein the server holds a pre-trained model to be fine-tuned; Multiple clients in federated learning hold private data corresponding to a target task to be fine-tuned; the apparatus comprises: A first fine-tuning unit is used to send the pre-trained model to be fine-tuned and the initial global prompt parameters for assisting the pre-trained model to perform the target task to a plurality of second computing devices corresponding to the plurality of clients, so that the plurality of second computing devices respectively perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the private data held by each of them, to obtain the respective private prompt parameters; a prompt parameter aggregation unit, configured to obtain a plurality of private prompt parameters obtained by the plurality of second computing devices, and aggregate the plurality of private prompt parameters to obtain updated global prompt parameters; A prompt parameter adjustment unit is used to calculate the similarity between the updated global prompt parameter and each private prompt parameter, and adjust the updated global prompt parameter based on the similarity; wherein the similarity between the adjusted global prompt parameter and each private prompt parameter is greater than the similarity between the updated global prompt parameter and each private prompt parameter.

13. A pre-training model fine-tuning device based on federated learning, characterized in that: Applied to any second computing device among a plurality of second computing devices corresponding to a plurality of clients in the federated learning; The plurality of clients hold private data corresponding to the target task to be fine-tuned; the device comprises: An acquisition unit, used to acquire a pre-trained model to be fine-tuned sent by a first computing device corresponding to a server in federated learning and an initial global prompt parameter for assisting the pre-trained model to perform the target task; A fine-tuning unit is used to perform prompt fine-tuning on the pre-trained model based on the initial global prompt parameters and the held private data to obtain private prompt parameters, and send the obtained private prompt parameters to the first computing device, so that the first computing device obtains the multiple private prompt parameters obtained by the multiple second computing devices, and aggregates the multiple private prompt parameters to obtain updated global prompt parameters, and calculates the similarity between the updated global prompt parameters and each private prompt parameter, and adjusts the updated global prompt parameters based on the similarity; wherein the similarity between the adjusted global prompt parameters and each private prompt parameter is greater than the similarity between the updated global prompt parameters and each private prompt parameter.

14. A computing device, characterized in that include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, the method according to any one of claims 1 to 10 or 11 is performed.

15. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 10 or 11 is implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 10 or 11 is implemented.

Images