Pre-training model fine tuning method, image classification method, medium, equipment and product

Abstract

The invention discloses a pre-training model fine tuning method, an image classification method, a medium, equipment and a product. The method comprises the following steps: training a first model; determining the sample loss of the first model obtained after training on each sample; executing shuffling and copy secret sharing of the first data set with each other participant, so that each participant respectively obtains two first fragments of the first data set obtained after shuffling; two second fragments of the model parameters of the pre-training model distributed by the model holder are received, multi-round iteration fine tuning is performed on the pre-training model with each other participant based on the two local first fragments, and in each round of iteration fine tuning, the two second fragments of the model parameters of the pre-training model distributed by the model holder are separated from each other participant; and the data holder determines K samples with the maximum RHO based on the two first fragments of the second data set and each other participant, and the K samples are utilized to perform back propagation to update model parameters. Valuable samples which are not learned yet are selected for model fine tuning, and the efficiency of safe fine tuning of the model is improved. The method can be applied to federated learning.

Description

Technical Field

[0001] The present disclosure relates to the field of cyberspace security technology, and in particular, to a pre-training model fine-tuning method, an image classification method, a medium, a device and a product. Background Art

[0002] Secure multi-party computing, also known as multi-party secure computing (MPC), can be used by multiple parties to jointly calculate the result of a function without disclosing the input data of each party in the function, and the result of the calculation is made public to one or more of the parties. Among them, secure fine-tuning of the model is a typical application of secure multi-party computing. The secure fine-tuning of the model usually includes two stages: the pre-trained model preparation stage and the stage of securely updating the model using fine-tuning samples. In the pre-trained model preparation stage, a large open source data set is usually used to perform multiple rounds of training on the general model in plain text to obtain a pre-trained model. In the stage of securely updating the model using fine-tuning samples, the model parameters of the fine-tuning samples and the pre-trained model are secretly shared, and multiple parties collaborate to perform multiple rounds of iterative fine-tuning to obtain the fine-tuned model. However, the secure fine-tuning scheme of this model has the problem of low efficiency. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for fine-tuning a pre-trained model, wherein the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, wherein the data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model, and the method is applied to the data holder, and the method includes: training a first model using at least part of the samples in the fine-tuning sample set; determining, for each sample in the first sample set, a first sample loss of the first model obtained after training on the sample, wherein the first sample set is a subset of the fine-tuning sample set; performing a shuffling and copying secret sharing operation of a first data set with the model holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; In response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the pre-trained model is fine-tuned for multiple rounds of iterations together with the model holder and the auxiliary computing party based on the two local first shards, so that each of the participating parties respectively performs multiple rounds of iterations of iterations of the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set together with the model holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0005] In a second aspect, the present disclosure provides a method for fine-tuning a pre-trained model, wherein the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, wherein the data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model, and the method is applied to the model holder, and the method includes: performing shuffling and copying secret sharing operations on a first data set with the data holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; the pre-trained model The model parameters are distributed to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants respectively obtains two second shards of the model parameters; based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds of iterations with the data holder and the auxiliary computing party, so that each of the participants respectively performs multiple rounds of iterations of iterations of iterations on the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0006] In a third aspect, the present disclosure provides a method for fine-tuning a pre-trained model, wherein the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, wherein the data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model. The method is applied to the auxiliary computing party, and the method includes: performing shuffling and copying secret sharing operations on a first data set with the data holder and the model holder, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is trained by the data holder using at least part of the samples in the fine-tuning sample set. In response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds with the data holder and the model holder, so that each of the participants respectively performs multiple rounds of iterative fine-tuning on the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the model holder, and uses the K samples to perform back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0007] In a fourth aspect, the present disclosure provides an image classification method, which is applied to a model holder, and the method includes: obtaining a first image to be classified; inputting the first image into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model using the fine-tuning method of the pre-trained model provided in the first to third aspects of the present disclosure, and the initial image classification model is a pre-trained model.

[0008] In a fifth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the third aspect of the present disclosure, or the steps of the method provided in the fourth aspect of the present disclosure.

[0009] In a sixth aspect, the present disclosure provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the third aspect of the present disclosure, or the steps of the method provided in the fourth aspect of the present disclosure.

[0010] In a seventh aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the third aspect of the present disclosure, or the steps of the method provided in the fourth aspect of the present disclosure.

[0011] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0012] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0014] Figure 1 The invention is a flowchart of a method for fine-tuning a pre-trained model applied to a data holder according to an exemplary embodiment.

[0015] Figure 2 It is a schematic diagram of a fine-tuning process of a pre-training model according to an exemplary embodiment.

[0016] Figure 3 The present invention is a flowchart of a method for fine-tuning a pre-trained model applied to a model holder according to an exemplary embodiment.

[0017] Figure 4 The present invention is a flowchart of a method for fine-tuning a pre-trained model applied to an auxiliary computing party according to an exemplary embodiment.

[0018] Figure 5 The figure is a flowchart of an image classification method according to an exemplary embodiment.

[0019] Figure 6 It is a block diagram of a fine-tuning device for a pre-trained model applied to a data holder according to an exemplary embodiment.

[0020] Figure 7 It is a block diagram of a fine-tuning device for a pre-trained model applied to a model holder according to an exemplary embodiment.

[0021] Figure 8 It is a block diagram of a fine-tuning device for a pre-trained model applied to an auxiliary computing party according to an exemplary embodiment.

[0022] Fig. 9 The figure is a block diagram of an image classification device according to an exemplary embodiment.

[0023] Fig.10 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] Before introducing the specific embodiments of the present disclosure, the terms involved in the present disclosure are first introduced and explained.

[0025] Secret Sharing is also called secret splitting or secret sharing. Its basic principle is to split secrets (such as keys, private data, etc.) into multiple shares (also called shards, Shares) and give them to different data parties for safekeeping. Only when parties exceeding a threshold number merge their shards can the secret be restored; Shares obtained from parties less than the threshold number cannot restore any information about the secret. In multi-party secure computing, the threshold number is usually the same as the number of participants, and the shares into which the secret is split can also be called shards.

[0026] Replicated Secret Sharing (RSS), also known as repetitive secret sharing, is an instance of secret sharing that is used to ensure that a secret value is shared among multiple entities and remains secure when any of them fails or is attacked. It involves three or more parties. In RSS, the original secret is divided into multiple parts and distributed to multiple entities. Each entity has a partial replica of the original secret, and the original secret value can only be restored when at least a certain number of entities exist at the same time. In this way, if one of the entities fails or is attacked, the other entities can still restore the original secret value through the replicas they each hold.

[0027] For example, in a three-party secure computing scenario, using replicated secret sharing, a secret value s can be split into three secret shares (also called replicated shards) e, g, and h, so that s = e + g + h, where the first party holds replicated shards e and g, the second party holds replicated shards g and h, and the third party holds replicated shards e and h. The secret value s can only be restored when any two of the three parties merge their replicated shards.

[0028] Two-Party Additive Secret Sharing (2PC-Additive) is a secret sharing scheme that allows two parties to recover the original secret value by adding their respective secret shares. In two-party additive secret sharing, the original secret value s is split into two pieces, which are assigned to the two parties respectively. Each party holds a piece of the original secret value, and the original secret value can only be recovered through the addition operation when both parties participate.

[0029] For example, in a two-party secure computing scenario, using replicated secret sharing, a secret value s can be split into two secret shares (also called shards) a and b, such that s = a + b, where the first party holds shard a and the second party holds shard b. The secret value s can only be restored when the two parties merge their shards.

[0030] Shuffling is a technology to protect data privacy, which allows participants to rearrange data without revealing their respective data. The main purpose of shuffling is to ensure the anonymity and untraceability of data, thereby enhancing privacy protection.

[0031] As discussed in the background technology, the secure fine-tuning of the model usually includes two stages: the pre-trained model preparation stage and the stage of securely updating the model using fine-tuning samples. In the pre-trained model preparation stage, a large open source data set is usually used to perform multiple rounds of training on the general model in plain text to obtain a pre-trained model. In the stage of securely updating the model using fine-tuning samples, the model parameters of the fine-tuning samples and the pre-trained model are secretly shared, and multiple parties collaborate to perform multiple rounds of iterative fine-tuning to obtain the fine-tuned model. Among them, for each round of iterative fine-tuning, all samples of the current round are used for back propagation to update the model parameters. These samples are not screened, resulting in a large number of iterative fine-tuning times, which affects the efficiency of secure fine-tuning of the pre-trained model.

[0032] In view of this, the present disclosure provides a pre-training model fine-tuning method, image classification method, medium, device and product.

[0033] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0036] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0039] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0040] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0041] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0042] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0043] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0044] Figure 1 FIG. 1 is a flow chart showing a method for fine-tuning a pre-trained model applied to a data holder according to an exemplary embodiment. Figure 2As shown, the participants in the fine-tuning of the pre-trained model include the data holder, the model holder, and the auxiliary computing party, wherein the data holder holds the fine-tuning sample set, and the model holder holds the pre-trained model. The fine-tuning samples in the fine-tuning sample set include a single data sample and the annotation results corresponding to the single data sample. The pre-trained model is usually obtained by training the general model in plain text using a large open source data set for multiple rounds. The pre-trained model can be a generation model (for example, a text generation model, an image generation model, etc.), a classification model (for example, a text classification model, an image classification model), a prediction model, and other types of models, which are not specifically limited in this disclosure.

[0045] For example, the pre-trained model is an initial image classification model, and the fine-tuning samples in the fine-tuning sample set include image samples and classification labels corresponding to the image samples.

[0046] like Figure 1 As shown, the fine-tuning method applied to the pre-trained model of the data holder may include S101 to S104.

[0047] In S101, the first model is trained using at least part of the samples in the fine-tuning sample set.

[0048] In the present disclosure, the first model can be trained using all samples in the fine-tuning sample set, or can be trained using some samples in the fine-tuning sample set, wherein the first model can be any model, for example, a neural network, specifically a residual network (Residual Network, ResNet). When the first model is trained using at least some samples in the fine-tuning sample set, the cutoff conditions for model training can be that the number of training times reaches a preset number of times, the model accuracy reaches a first preset accuracy threshold, the model loss is less than a first preset loss threshold, etc.

[0049] In S102, for each sample in the first sample set, a first sample loss of the first model obtained after training on the sample is determined, wherein the first sample set is a subset of the fine-tuning sample set.

[0050] In the present disclosure, the first model obtained after the above-mentioned S101 training can be used as an irreducible loss (IL) model. Afterwards, for each sample in the first sample set, the sample can be input into the IL model to obtain a predicted value of the IL model. Then, based on the difference between the predicted value and the labeling result corresponding to the sample, the first sample loss of the IL model on the sample is calculated as the IL of the sample.

[0051] In one implementation, the first sample set is a fine-tuning sample set, that is, the IL of each sample in the fine-tuning sample set is calculated.

[0052] In another embodiment, the first sample set is a proper subset of the fine-tuning sample set. For example, the first sample set is a difference set between the fine-tuning sample set and the sample set used for training the first model in the above S101.

[0053] In S103, shuffling and replicating secret sharing operations of the first data set are performed with the model holder and the auxiliary computing party, so that each participant obtains two first fragments of the first data set obtained after shuffling, and the first data set includes a first sample set and a first sample loss of each sample in the first sample set.

[0054] In S104, in response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the pre-trained model is fine-tuned for multiple rounds of iterations together with the model holder and the auxiliary computing party based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards respectively until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set together with the model holder and the auxiliary computing party, and uses the K samples to perform back propagation to update the model parameters.

[0055] In the present disclosure, the second sample set is samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0056] like Figure 2 As shown, after obtaining the IL of each sample in the first sample set, the three participants can securely shuffle the first sample set and the IL of each sample in the first sample set (hereinafter referred to as the first data set). Specifically, the data holder, the model holder and the auxiliary computing party can collaboratively perform the shuffling and copy secret sharing operations of the first data set, so that each participant can obtain two first shards of the first data set obtained after shuffling. The fine-tuning samples held by the data holder are distributed to the three participants after shuffling and copy secret sharing, which can not only avoid the leakage of fine-tuning samples, but also avoid the data holder from inferring the selected samples based on the correspondence between the selected sample shards (i.e., the shards of the K samples with the largest RHO) and the fine-tuning samples, and then infer the model information of the pre-trained model, thereby avoiding the leakage of model information.

[0057] In order to avoid the leakage of model parameters (i.e., model weights), the model owner can distribute the model parameters of the pre-trained model to the data owner and auxiliary computing party (such as Figure 2As shown), so that each participant obtains two second slices of model parameters respectively. Specifically, the model holder can distribute the model parameters to the data holder and the auxiliary computing party in the following way in a copy secret sharing manner: first, the model holder randomly generates the first matrix and the second matrix of the same dimension as the parameter matrix locally, where the parameter matrix is ​​a matrix composed of the model parameters of the pre-trained model; then, the model holder subtracts the first matrix and the second matrix from the parameter matrix in sequence to obtain the third matrix, that is, the third matrix = parameter matrix - first matrix - second matrix; then, the model holder sends the first matrix and the third matrix to the data holder, and the data holder uses them as the two second slices of model parameters after receiving the first matrix and the third matrix sent by the model holder. At the same time, the model holder sends the second matrix and the third matrix to the auxiliary computing party, and the auxiliary computing party uses them as the two second slices of model parameters after receiving the second matrix and the third matrix sent by the model holder; at the same time, the model holder uses the first matrix and the second matrix as the two second slices of local model parameters, so that the model parameters can be distributed to the three participants in the fine-tuning of the pre-trained model in a copy secret sharing manner.

[0058] After receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the data holder can collaborate with the model holder and the auxiliary computing party to perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two first shards of the shuffled first data set held by itself, so that each participant performs multiple rounds of iterative fine-tuning on the two second shards of the model parameters held by itself, until the fine-tuning cutoff condition is reached. When iteratively fine-tuning the pre-trained model, the first sample set can be input into the pre-trained model in batches to select K samples with the largest reducible holdout loss (RHO) from the input samples of the current batch according to the output results of the pre-trained model, and then back-propagate the model parameters using the second sample losses corresponding to each of the selected K samples to complete one iterative fine-tuning, where K≥1.

[0059] RHO means the difference between the loss of the current sample (i.e., the loss of the second sample) and the loss of the sample when the model is trained. The larger the RHO, the more the sample needs to be learned. Therefore, K samples with the largest RHO are selected from the input samples of the current batch to fine-tune the model parameters.

[0060] Specifically, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two first slices of the first data set obtained after shuffling, specifically, the three participants respectively perform multiple rounds of iterative fine-tuning on the two second slices of the model parameters held by themselves, until the fine-tuning cutoff condition is reached. The fine-tuning cutoff condition may be that the accuracy of the model reaches the second accuracy threshold, the model loss is less than the second preset loss threshold, etc.

[0061] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0062] The following is a detailed description of a specific implementation method for the data holder, the model holder, and the auxiliary computing party in the above S103 to perform the shuffling and copying secret sharing operations on the first data set so that each participant obtains two first fragments of the first data set obtained after shuffling. Specifically, it can be achieved through the following steps (a1) and (a2).

[0063] Step (a1): The data holder may distribute the first data set to the model holder and the auxiliary computing party in a two-party additive secret sharing manner, so that the model holder and the auxiliary computing party respectively obtain a fifth shard of the first data set, and respectively use the first random seed to shuffle the fifth shards held by themselves, generate two first shards of the shuffled first data set based on the shuffled fifth shards, and send the first shard of the two first shards that is not shared by the model holder and the auxiliary computing party to the data holder.

[0064] Step (a2): The data holder receives the first fragments respectively sent by the model holder and the auxiliary computing party as the two first fragments of the first data set obtained after shuffling.

[0065] Specifically, the data holder can distribute the first data set to the model holder and the auxiliary computing party in a two-party addition secret sharing manner in the following manner:

[0066] First, the data holder can locally generate a third random data set with the same specifications as the first data set; then, the data holder subtracts the data at the corresponding positions of the first data set and the third random data set to obtain the fifth data set; finally, the data holder sends the third random data set and the fifth data set as two shards of the first data set to the model holder and the auxiliary computing party respectively. For example, the third random data set is sent to the model holder. After receiving the third random data set, the model holder uses the third random data set as a shard of the first data set (called the fifth shard). At the same time, the fifth data set is sent to the auxiliary computing party. After receiving the fifth data set, the auxiliary computing party can use the fifth data set as another shard of the first data (called the fifth shard).

[0067] The same specifications of two data sets refer to the consistency of the two data sets in structure and format. Specifically, it can include the following aspects:

[0068] Dimensions of the dataset: Both datasets have the same number of rows and columns;

[0069] Data type: The data at corresponding positions have the same data type, such as integers, floating-point numbers, or strings;

[0070] Data structure: The data in both datasets are organized in the same way, for example, both are in tabular form;

[0071] Data range: The data values ​​in two data sets may come from the same value range, for example, both are integers between 0 and 100.

[0072] The third random data set has the same specifications as the first data set, which means that the dimension, data type, data structure and data range of the third random data set are the same as those of the first data set.

[0073] Specifically, the model holder and the auxiliary computing party can achieve shuffling and copying secret sharing of the first data set in the following way:

[0074] After receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, the model holder can shuffle the fifth shard using the first random seed to obtain a sixth shard; then, the model holder generates a first random data set with the same specifications as the sixth shard, and sends the first random data set to the auxiliary computing party; next, the model holder subtracts the data at the corresponding positions of the sixth shard from the first random data set to obtain a third data set, determines the third data set as a shard of the first data set obtained after shuffling, and sends the third data set as a shard of the first data set obtained after shuffling to the data holder. The first random data set has the same specifications as the sixth shard, which means that the dimension, data type, data structure, and data range of the first random data set are the same as those of the sixth shard, and are essentially the same specifications as the first data set.

[0075] At the same time, after receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, the auxiliary computing party shuffles the fifth shard using the first random seed to obtain the seventh shard; then, the auxiliary computing party generates a second random data set with the same specifications as the seventh shard, and sends the second random data set to the model holder; next, the auxiliary computing party subtracts the data at the corresponding positions of the seventh shard from the second random data set to obtain the fourth data set, determines the fourth data set as a shard of the first data set obtained after shuffling, and sends the fourth data set as a shard of the first data set obtained after shuffling to the data holder. The second random data set has the same specifications as the seventh shard, which means that the dimension, data type, data structure and data range of the second random data set are the same as those of the seventh shard, which is essentially the same as the specifications of the first data set.

[0076] After receiving the second random data set sent by the auxiliary computing party, the model holder can determine the sum of the locally generated first random data set and the second random data set received from the auxiliary computing party as another shard of the first data set obtained after shuffling. That is, after the shuffling and copy secret sharing operations of the first data set, the model holder can hold two shards of the first data set obtained after shuffling, namely: the third data set and the sum of the first random data set and the second random data set.

[0077] After receiving the first random data set sent by the model holder, the auxiliary computing party may determine the sum of the first random data set received from the model holder and the second random data set generated locally as another shard of the first data set obtained after shuffling. That is, after the shuffling and copying secret sharing operations of the first data set, the auxiliary computing party may hold two shards of the first data set obtained after shuffling, namely: the fourth data set and the sum of the first random data set and the second random data set.

[0078] After receiving the third data set sent by the model holder and the fourth data set sent by the auxiliary computing party, the data holder uses them as the two first fragments of the first data set obtained after shuffling. After the shuffling and copying secret sharing operations of the first data set, the data holder can hold two fragments of the first data set obtained after shuffling, namely: the third data set and the fourth data set.

[0079] The first random seed is a random seed shared by the auxiliary computing party and the model holder. The model holder and the auxiliary computing party use the same random seed (i.e., the first random seed) to shuffle the slices of the first data set held by each party, which can ensure that the data holder and the auxiliary computing party have the same order of shuffling the slices of the data set held by themselves, thereby ensuring the accuracy of subsequent model fine-tuning calculations. It should be noted that the specific method of shuffling the data set using a random seed is well known to those skilled in the art and will not be repeated in this disclosure.

[0080] The following is a detailed description of the specific implementation method of the data holder in the above S104, based on the two first shards of the local second data set, together with the model holder and the auxiliary calculation party, determining the K samples with the largest reduction in retention loss RHO from the second sample set. Specifically, it can be achieved through the following steps (b1) to (b3).

[0081] Step (b1): The data holder calculates the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set based on the two shards of the local second sample set together with the model holder and the auxiliary computing party, so that each participant obtains two third shards of the second sample loss.

[0082] In the present disclosure, the three participants securely calculate the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set based on the two shards of the local second sample set. Specifically, Figure 2 As shown, the three participants can use the secure addition protocol, secure multiplication protocol, secure comparison protocol and truncation protocol to perform secure forward propagation on the samples used for this round of iterative fine-tuning (i.e., the second sample set), and calculate the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set, so that the three participants can respectively obtain two third slices of the second sample loss of each sample in the second sample set.

[0083] Step (b2): The data holder calculates the RHO of each sample in the second sample set with the model holder and the auxiliary calculation party based on the two shards of the first sample loss of each sample in the local second sample set and the two local third shards, so that each participant obtains the two fourth shards of the RHO of each sample in the second sample set.

[0084] In the present disclosure, the RHO of a sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample. After the three participants have obtained the two third slices of the second sample loss of each sample in the second sample set, the three participants can securely calculate the RHO of each sample in the second sample set based on the two slices of the first sample loss of each sample in the second sample set and the two third slices of the second sample loss of each sample in the second sample set held by the three participants, so that each participant obtains the two fourth slices of the RHO of each sample in the second sample set.

[0085] Specifically, for each sample in the second sample set, the three participants can securely calculate the RHO of each sample in the second sample set based on the two shards lost by the first sample of the sample and the two third shards lost by the second sample of the sample held by themselves, so that each participant can obtain the two fourth shards of the RHO of the sample.

[0086] Step (b3): ​​The data holder, together with the model holder and the auxiliary calculation party, determines K samples with the largest RHO from the second sample set based on the two fourth slices of the RHO of each sample in the local second sample set.

[0087] In a possible implementation, the data holder can sort the RHO of each sample in the local second sample set through a multi-party secure sorting algorithm with the model holder and the auxiliary computing party based on the two fourth shards of the RHO of each sample in the second sample set, and determine the K samples with the largest RHO from the second sample set according to the sorting result. It should be noted that the specific implementation of the multi-party secure sorting algorithm is well known to those skilled in the art and will not be described in detail in this disclosure.

[0088] For example, Figure 2 As shown, the second sample set includes sample x2, sample x0 and sample x1, which are input into the pre-trained model obtained after the previous round of iterative fine-tuning, and secure forward propagation is performed to calculate the second sample losses of the pre-trained model obtained after the previous round of iterative fine-tuning on sample x2, sample x0 and sample x1, respectively, to obtain the second sample loss loss2 of sample x2, the second sample loss loss0 of sample x0, and the second sample loss loss1 of sample x1; the first sample loss of each sample in the second sample set obtained by the above S102 is as follows: the first sample loss of sample x2 is IL loss2, the first sample loss of sample x0 is IL loss0, and the first sample loss of sample x1 is IL loss1, thereby, the RHO of each sample can be calculated, that is, the RHO of sample x2 is RHO loss2, the RHO of sample x0 is RHO loss0, and the RHO of sample x1 is RHOloss1; the sorting results obtained by sorting these RHOs by the multi-party secure sorting algorithm are: RHO loss2 (the sequence number of the sorting result is 0), RHO loss1 (the sequence number of the sorting result is 1), RHO loss0, if K=2, the K samples with the largest RHO include: sample x2 corresponding to RHO loss2, sample x1 corresponding to RHO loss1; finally, the second sample loss loss2 of sample x2 and the second sample loss loss2 of sample x1 are used to perform back propagation to update the model parameters.

[0089] Figure 3 The flowchart of a method for fine-tuning a pre-trained model applied to a model holder according to an exemplary embodiment is shown. The participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds a pre-trained model. Figure 3 As shown, the fine-tuning method of the pre-trained model applied to the model holder may include the following S201 to S203.

[0090] In S201, shuffling and replicating secret sharing operations of the first data set are performed with the data holder and the auxiliary computing party, so that each participant obtains two first fragments of the first data set obtained after shuffling, and the first data set includes a first sample set and a first sample loss of each sample in the first sample set, and the first sample set is a subset of the fine-tuning sample set.

[0091] Among them, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder by training the first model using at least part of the samples in the fine-tuning sample set.

[0092] In S202, the model parameters of the pre-trained model are distributed to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters respectively.

[0093] In S203, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds of iterations together with the data holder and the auxiliary computing party, so that each participant performs multiple rounds of iterations of iterations on the two local second shards respectively until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines, based on the two first shards of the local second data set, together with the data holder and the auxiliary computing party, K samples with the largest reducible retention loss RHO from the second sample set, and uses the K samples for back propagation to update the model parameters.

[0094] The second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0095] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0096] Optionally, the model holder, based on the two first shards of the local second data set, determines, together with the data holder and the auxiliary computing party, K samples with the largest RHO that can be reduced from the second sample set, including: the model holder, together with the data holder and the auxiliary computing party, calculates, based on the two shards of the local second sample set, the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set, so that each participant obtains two third shards of the second sample loss; the model holder, together with the data holder and the auxiliary computing party, calculates the RHO of each sample in the second sample set according to the two shards of the first sample loss of each sample in the local second sample set and the two local third shards, so that each participant obtains two fourth shards of the RHO of each sample in the second sample set, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; the model holder, together with the data holder and the auxiliary computing party, determines, from the second sample set, K samples with the largest RHO, based on the two fourth shards of the RHO of each sample in the local second sample set.

[0097] Optionally, the model holder, based on the two fourth shards of the RHO of each sample in the local second sample set, works with the data holder and the auxiliary computing party to determine K samples with the largest RHO from the second sample set, including: the model holder, based on the two fourth shards of the RHO of each sample in the local second sample set, works with the data holder and the auxiliary computing party to sort the RHO of each sample in the second sample set through a multi-party secure sorting algorithm, and determines K samples with the largest RHO from the second sample set according to the sorting results.

[0098] Optionally, the above S201 may include: in response to receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, shuffling the local fifth shard using a first random seed to obtain a sixth shard; generating a first random data set with the same specifications as the sixth shard, and sending the first random data set to the auxiliary computing party; subtracting the data at corresponding positions of the sixth shard from the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holder; in response to receiving a second random data set sent by the auxiliary computing party, determining the sum of the first random data set and the second random data set as the first shard.

[0099] In addition, if Figure 2As shown, after completing the fine-tuning of the pre-trained model, the three participants can respectively obtain two second shards of the model parameters of the pre-trained model obtained after fine-tuning. In order to restore the pre-trained model obtained after fine-tuning, the model holder can obtain the second shard obtained after fine-tuning and not held by the model holder from the data holder or the auxiliary computing party, and merge the three shards of the pre-trained model obtained after fine-tuning to obtain the model parameters of the pre-trained model obtained after fine-tuning, so as to restore the pre-trained model obtained after fine-tuning. Specifically, the above-mentioned fine-tuning method of the pre-trained model applied to the model holder can also include the following two steps: obtaining the second shard obtained after fine-tuning and not held by the model holder from the data holder or the auxiliary computing party; merging the second shard obtained from the data holder or the auxiliary computing party with the two local second shards to obtain the model parameters of the pre-trained model obtained after fine-tuning.

[0100] The specific implementation methods of each step in the fine-tuning method of the pre-trained model applied to the model holder according to the embodiment of the present disclosure have been described in detail in the fine-tuning method of the pre-trained model applied to the data holder according to the embodiment of the present disclosure, and will not be repeated here.

[0101] Figure 4 The flowchart of a method for fine-tuning a pre-trained model applied to an auxiliary computing party according to an exemplary embodiment is shown. The parties involved in fine-tuning the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds a pre-trained model. Figure 4 As shown, the fine-tuning method of the pre-trained model applied to the auxiliary computing party may include the following S301 and S302.

[0102] In S301, shuffling and replicating secret sharing operations of a first data set are performed with a data holder and a model holder, so that each participant obtains two first fragments of the first data set obtained after shuffling, the first data set includes a first sample set and a first sample loss of each sample in the first sample set, and the first sample set is a subset of the fine-tuning sample set.

[0103] Among them, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder by training the first model using at least part of the samples in the fine-tuning sample set.

[0104] In S302, in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the pre-trained model is fine-tuned for multiple rounds of iterations together with the data holder and the model holder based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards respectively until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, together with the data holder and the model holder, K samples with the largest reducible retention loss RHO from the second sample set based on the two first shards of the local second data set, and uses the K samples for back propagation to update the model parameters.

[0105] The second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0106] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0107] Optionally, the auxiliary computing party determines, together with the data holder and the model holder, K samples with the largest RHO that can be reduced from the second sample set based on the two first shards of the local second data set, including: the auxiliary computing party calculates, together with the data holder and the model holder, the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set based on the two shards of the local second sample set, so that each participant obtains two third shards of the second sample loss; the auxiliary computing party calculates, together with the data holder and the model holder, the RHO of each sample in the second sample set based on the two shards of the first sample loss of each sample in the local second sample set and the two local third shards, so that each participant obtains two fourth shards of the RHO of each sample in the second sample set, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; the auxiliary computing party determines, together with the data holder and the model holder, K samples with the largest RHO from the second sample set based on the two fourth shards of the RHO of each sample in the local second sample set.

[0108] Optionally, the auxiliary computing party determines K samples with the largest RHO from the second sample set based on the two fourth shards of the RHO of each sample in the local second sample set, together with the data holder and the model holder, including: the auxiliary computing party sorts the RHO of each sample in the second sample set through a multi-party secure sorting algorithm based on the two fourth shards of the RHO of each sample in the local second sample set, together with the data holder and the model holder, and determines K samples with the largest RHO from the second sample set according to the sorting results.

[0109] Optionally, the above S301 may include: in response to receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, shuffling the local fifth shard using a first random seed to obtain a seventh shard; generating a second random data set with the same specifications as the seventh shard, and sending the second random data set to the model holder; subtracting the data at corresponding positions of the seventh shard from the second random data set to obtain a fourth data set, determining the fourth data set as the first shard, and sending the fourth data set as the first shard to the data holder; in response to receiving the first random data set sent by the model holder, determining the sum of the first random data set and the second random data set as the first shard.

[0110] The specific implementation methods of each step in the fine-tuning method of the pre-trained model applied to the auxiliary computing party according to the embodiment of the present disclosure have been described in detail in the fine-tuning method of the pre-trained model applied to the data holder according to the embodiment of the present disclosure, and will not be repeated here.

[0111] Figure 5is a flowchart of an image classification method according to an exemplary embodiment. The image classification method can be applied to a model holder, such as Figure 5 As shown, the image classification method may include the following S401 and S402.

[0112] In S401, a first image to be classified is obtained.

[0113] In the present disclosure, the first image may be any type of image, for example, a medical image, and the present disclosure does not make any specific limitation.

[0114] In S402, the first image is input into a target image classification model to obtain a classification result of the first image.

[0115] In the present disclosure, the target image classification model is obtained by fine-tuning the initial image classification model according to the fine-tuning method of the above-mentioned pre-trained model provided in the present disclosure, wherein the initial image classification model is a pre-trained model.

[0116] By fine-tuning the initial image classification model using the fine-tuning method of the above-mentioned pre-trained model provided in the present disclosure, the fine-tuning efficiency of the initial image classification model can be improved.

[0117] Figure 6 1 is a block diagram of a device for fine-tuning a pre-trained model applied to a data holder according to an exemplary embodiment. The parties involved in fine-tuning the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model. Figure 6As shown, the fine-tuning device 500 for the pre-trained model applied to the data holder includes: a training module 501, which is used to train the first model using at least part of the samples in the fine-tuning sample set; a determination module 502, which is used to determine, for each sample in the first sample set, a first sample loss of the first model obtained after training on the sample, wherein the first sample set is a subset of the fine-tuning sample set; a first secure shuffling module 503, which is used to perform shuffling and copy secret sharing operations on the first data set with the model holder and the auxiliary computing party, so that each of the participating parties respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; a first secure fine-tuning module 504, which is used to respond to the receipt of the model holder's copy secret sharing operation. The two second shards of the model parameters of the pre-trained model distributed in a secret sharing manner, based on the two local first shards, perform multiple rounds of iterative fine-tuning on the pre-trained model with the model holder and the auxiliary computing party, so that each of the participating parties respectively performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the model holder and the auxiliary computing party, and uses the K samples to perform back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0118] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0119] Optionally, the first security fine-tuning module 504 includes: a first calculation submodule, which is used to calculate, based on two slices of the local second sample set, the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set with the model holder and the auxiliary calculation party, so that each of the participants obtains two third slices of the second sample loss respectively; a second calculation submodule, which is used to calculate, based on the two slices of the first sample loss of each sample in the local second sample set and the two third slices of the local first sample loss, the RHO of each sample in the second sample set respectively, with the model holder and the auxiliary calculation party, so that each of the participants obtains two fourth slices of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; a first determination submodule, which is used to determine, based on the two fourth slices of the RHO of each sample in the local second sample set, with the model holder and the auxiliary calculation party, the K samples with the largest RHO from the second sample set.

[0120] Optionally, the first determination submodule is used to sort the RHO of each sample in the second sample set through a multi-party secure sorting algorithm based on the two fourth shards of the RHO of each sample in the local second sample set, together with the model holder and the auxiliary calculation party, and determine K samples with the largest RHO from the second sample set according to the sorting results.

[0121] Optionally, the first secure shuffling module 503 includes: a secret sharing submodule, used to distribute the first data set to the model holder and the auxiliary computing party in a two-party additive secret sharing manner, so that the model holder and the auxiliary computing party respectively obtain a fifth shard of the first data set, and respectively use the first random seed to shuffle the fifth shard held by themselves, generate two first shards of the shuffled first data set based on the shuffled fifth shard, and send the first shard of the two first shards that is not shared by the model holder and the auxiliary computing party to the data holder; a receiving submodule, used to receive the first shards sent by the model holder and the auxiliary computing party respectively as the two first shards of the shuffled first data set.

[0122] Figure 7 The block diagram of a fine-tuning device for a pre-trained model applied to a model holder according to an exemplary embodiment is shown. The parties involved in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model. Figure 7As shown, the fine-tuning device 600 for the pre-trained model applied to the model holder includes: a second secure shuffling module 601, which is used to perform shuffling and copy secret sharing operations on the first data set with the data holder and the auxiliary computing party, so that each of the participating parties respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; a model parameter copy secret sharing module 602, which is used to distribute the model parameters of the pre-trained model to the data holder and the auxiliary computing party in a copy secret sharing manner. The method comprises the following steps: a first security fine-tuning module 601 is used to perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two first slices of the local data, together with the data holder and the auxiliary computing party, so that each of the participants respectively obtains the two second slices of the model parameters; a second security fine-tuning module 603 is used to perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two first slices of the local data, together with the data holder and the auxiliary computing party, until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines, based on the two first slices of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set, together with the data holder and the auxiliary computing party, and uses the K samples to perform back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0123] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0124] Optionally, the second security fine-tuning module 603 includes: a third calculation submodule, which is used to calculate, based on two slices of the local second sample set, the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set with the data holder and the auxiliary calculation party, so that each of the participants obtains two third slices of the second sample loss respectively; a third calculation submodule, which is used to calculate, based on the two slices of the first sample loss of each sample in the local second sample set and the two local third slices, the RHO of each sample in the second sample set respectively, with the data holder and the auxiliary calculation party, so that each of the participants obtains two fourth slices of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; a second determination submodule, which is used to determine, based on the two fourth slices of the RHO of each sample in the local second sample set, with the data holder and the auxiliary calculation party, the K samples with the largest RHO from the second sample set.

[0125] Optionally, the second determination submodule is used to sort the RHO of each sample in the second sample set through a multi-party secure sorting algorithm based on the two fourth shards of the RHO of each sample in the local second sample set, together with the data holder and the auxiliary computing party, and determine K samples with the largest RHO from the second sample set according to the sorting results.

[0126] Optionally, the second security shuffling module 601 includes: a first shuffling submodule, used to, in response to receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, shuffle the local fifth shard using a first random seed to obtain a sixth shard; a first generation submodule, used to generate a first random data set with the same specification as the sixth shard, and send the first random data set to the auxiliary computing party; a third determination submodule, used to subtract the data at corresponding positions of the sixth shard from the first random data set to obtain a third data set, determine the third data set as the first shard, and send the third data set as the first shard to the data holder; a fourth determination submodule, used to, in response to receiving a second random data set sent by the auxiliary computing party, determine the sum of the first random data set and the second random data set as the first shard.

[0127] Optionally, the fine-tuning device 600 for the pre-trained model applied to the model holder also includes: a first acquisition module, used to obtain the second slice obtained after fine-tuning and not held by the model holder from the data holder or the auxiliary computing party; a merging module, used to merge the second slice obtained from the data holder or the auxiliary computing party with the two local second slices to obtain model parameters of the pre-trained model obtained after fine-tuning.

[0128] Figure 8 The block diagram of a fine-tuning device for a pre-trained model applied to an auxiliary computing party according to an exemplary embodiment is shown. The parties involved in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model. Figure 8 As shown, the fine-tuning device 700 of the pre-trained model applied to the auxiliary computing party includes: a third secure shuffling module 701, which is used to perform shuffling and copy secret sharing operations on the first data set with the data holder and the model holder, so that each of the participating parties respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; a third secure fine-tuning module 702, which is used to respond to the receipt of the model holder's request in the form of copy secret sharing. The two second shards of the model parameters of the pre-trained model distributed are based on the two local first shards, and the data holder and the model holder perform multiple rounds of iterative fine-tuning on the pre-trained model, so that each of the participants respectively performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the model holder, and uses the K samples to perform back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0129] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, the first sample loss of the first model obtained after training on each sample in the first sample set is determined; thereafter, the three participants in the model fine-tuning jointly perform the shuffling and replication secret sharing operations of the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two local first shards, so that each participant performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached. Among them, in each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the local second data set, and use these K samples to back-propagate and update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, each round of iterative fine-tuning uses the K samples with the largest RHO to update the model parameters. In this way, those learnable, valuable and unlearned samples can be selected for model fine-tuning, reducing the waste of calculations on redundant or noise samples, thereby reducing the number of iterative fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid the leakage of model parameters. In addition, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and copying secret sharing, which can not only prevent the leakage of fine-tuning samples, but also prevent the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and then inferring the model information of the pre-trained model, thereby avoiding the leakage of model information. This method can be applied to federated learning.

[0130] Optionally, the third security shuffling module 701 includes: a second shuffling submodule, used to, in response to receiving a fifth slice of the first data set distributed by the data holder in a two-party addition secret sharing manner, shuffle the local fifth slice using a first random seed to obtain a seventh slice; a second generation submodule, used to generate a second random data set with the same specification as the seventh slice, and send the second random data set to the model holder; a fifth determination submodule, used to subtract the data at corresponding positions of the seventh slice from the second random data set to obtain a fourth data set, determine the fourth data set as the first slice, and send the fourth data set as the first slice to the data holder; a sixth determination submodule, used to, in response to receiving the first random data set sent by the model holder, determine the sum of the first random data set and the second random data set as the first slice.

[0131] Fig. 9 is a block diagram of an image classification device according to an exemplary embodiment. The image classification device is applied to a model holder, such as Fig. 9 As shown, the image classification device 800 includes: a second acquisition module 801, used to acquire a first image to be classified; a classification module 802, used to input the first image into an image classification model to obtain a classification result of the first image, wherein the first image is input into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of the above-mentioned pre-trained model provided in the present disclosure, and the initial image classification model is a pre-trained model.

[0132] By fine-tuning the initial image classification model using the fine-tuning method of the above-mentioned pre-trained model provided in the present disclosure, the fine-tuning efficiency of the initial image classification model can be improved.

[0133] Reference below Fig.10 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 900 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0134] like Fig.10As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0135] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0136] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0137] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0138] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0139] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0140] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: trains the first model using at least part of the samples in the fine-tuning sample set, where the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, where the data holder holds the fine-tuning sample set, and the model holder holds the pre-training model; determines, for each sample in the first sample set, a first sample loss of the first model obtained after training on the sample, wherein the first sample set is a subset of the fine-tuning sample set; performs shuffling and copying secret sharing operations on the first data set with the model holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set ; In response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds with the model holder and the auxiliary computing party, so that each of the participants respectively performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set together with the model holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0141] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: performs shuffling and copying secret sharing operations on the first data set with the data holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the participants in the fine-tuning of the pre-trained model include the data holder, the model holder and the auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; The model parameters of the training model are distributed to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants obtains two second shards of the model parameters respectively; based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds of iterations with the data holder and the auxiliary computing party, so that each of the participants performs multiple rounds of iterative fine-tuning on the two local second shards respectively until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0142] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: performs shuffling and copying secret sharing operations on the first data set with the data holder and the model holder, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the participants in the fine-tuning of the pre-trained model include the data holder, the model holder and the auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is calculated by the data holder using at least part of the samples in the fine-tuning sample set to calibrate the first model. model is trained; in response to receiving two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds of iterations with the data holder and the model holder, so that each of the participants respectively performs multiple rounds of iterations of iterations of the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the model holder, and uses the K samples to perform back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the first acquisition module may also be described as "a module that obtains the second shard obtained after fine-tuning from the data holder or the auxiliary computing party and not held by the model holder".

[0146] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] According to one or more embodiments of the present disclosure, Example 1 provides a method for fine-tuning a pre-trained model, wherein the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, wherein the data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model, and the method is applied to the data holder, and the method includes: training a first model using at least part of the samples in the fine-tuning sample set; for each sample in the first sample set, determining a first sample loss of the first model obtained after training on the sample, wherein the first sample set is a subset of the fine-tuning sample set; performing shuffling and copying secret sharing operations on the first data set with the model holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first fragments of each sample in the first sample set. Sample loss; in response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds with the model holder and the auxiliary computing party, so that each of the participants respectively performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the model holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0149] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the two first shards of the local second data set, together with the model holder and the auxiliary computing party, determine K samples with the largest RHO that can be reduced from the second sample set, including: based on the two shards of the local second sample set, together with the model holder and the auxiliary computing party, calculate the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set, so that each of the participants obtains two third shards of the second sample loss respectively; based on the two shards of the first sample loss of each sample in the local second sample set and the two third shards of the local, calculate the RHO of each sample in the second sample set respectively with the model holder and the auxiliary computing party, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; based on the two fourth shards of the RHO of each sample in the local second sample set, together with the model holder and the auxiliary computing party, determine K samples with the largest RHO from the second sample set.

[0150] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the two fourth shards of the RHO of each sample in the second sample set based on the local data, and the model holder and the auxiliary computing party determine the K samples with the largest RHO from the second sample set, including: based on the two fourth shards of the RHO of each sample in the second sample set based on the local data, and the model holder and the auxiliary computing party sort the RHO of each sample in the second sample set through a multi-party secure sorting algorithm, and determine the K samples with the largest RHO from the second sample set according to the sorting result.

[0151] According to one or more embodiments of the present disclosure, Example 4 provides a method of any one of Examples 1-3, wherein the shuffling and replication secret sharing operations of the first data set are performed with the model holder and the auxiliary computing party so that each of the participating parties respectively obtains two first shards of the shuffled first data set, including: distributing the first data set to the model holder and the auxiliary computing party in a two-party addition secret sharing manner so that the model holder and the auxiliary computing party respectively obtain a fifth shard of the first data set, and shuffling the fifth shards held by themselves using a first random seed respectively, generating two first shards of the shuffled first data set based on the shuffled fifth shard, and sending the first shard of the two first shards that is not shared by the model holder and the auxiliary computing party to the data holder; receiving the first shards sent by the model holder and the auxiliary computing party respectively as the two first shards of the shuffled first data set.

[0152] According to one or more embodiments of the present disclosure, Example 5 provides a method for fine-tuning a pre-trained model, wherein the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, wherein the data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model, and the method is applied to the model holder, and the method includes: performing shuffling and copying secret sharing operations on a first data set with the data holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding samples, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; The model parameters of the pre-trained model are distributed to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants obtains two second shards of the model parameters respectively; based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds of iterations with the data holder and the auxiliary computing party, so that each of the participants performs multiple rounds of iterative fine-tuning on the two local second shards respectively until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0153] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, wherein the two first shards of the local second data set, together with the data holder and the auxiliary computing party, determine K samples with the largest RHO that can be reduced from the second sample set, including: based on the two shards of the local second sample set, together with the data holder and the auxiliary computing party, calculate the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set, so that each of the participants obtains two third shards of the second sample loss; based on the two shards of the first sample loss of each sample in the local second sample set and the two third shards, together with the data holder and the auxiliary computing party, calculate the RHO of each sample in the second sample set, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; based on the two fourth shards of the RHO of each sample in the local second sample set, together with the data holder and the auxiliary computing party, determine K samples with the largest RHO from the second sample set.

[0154] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 6, wherein the two fourth shards of the RHO of each sample in the second sample set based on the local data, and the data holder and the auxiliary computing party determine the K samples with the largest RHO from the second sample set, including: based on the two fourth shards of the RHO of each sample in the second sample set based on the local data, and the data holder and the auxiliary computing party sort the RHO of each sample in the second sample set through a multi-party secure sorting algorithm, and determine the K samples with the largest RHO from the second sample set according to the sorting result.

[0155] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 5, which performs shuffling and copying secret sharing operations of the first data set with the data holder and the auxiliary computing party so that each of the participants respectively obtains two first shards of the first data set obtained after shuffling, including: in response to receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, shuffling the local fifth shard using a first random seed to obtain a sixth shard; generating a first random data set with the same specification as the sixth shard, and sending the first random data set to the auxiliary computing party; subtracting the data at corresponding positions of the sixth shard from the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holder; in response to receiving a second random data set sent by the auxiliary computing party, determining the sum of the first random data set and the second random data set as the first shard.

[0156] According to one or more embodiments of the present disclosure, Example 9 provides the method of any one of Examples 5-8, which further includes: obtaining the second shard obtained after fine-tuning and not held by the model holder from the data holder or the auxiliary computing party; merging the second shard obtained from the data holder or the auxiliary computing party with the two local second shards to obtain model parameters of the pre-trained model obtained after fine-tuning.

[0157] According to one or more embodiments of the present disclosure, Example 10 provides a method for fine-tuning a pre-trained model, wherein the participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party, wherein the data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model, and the method is applied to the auxiliary computing party, and the method includes: performing shuffling and copying secret sharing operations on a first data set with the data holder and the model holder, so that each of the participants respectively obtains two first slices of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is shuffled by the data holder using at least part of the samples in the fine-tuning sample set. A model is trained; in response to receiving two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds of iterations with the data holder and the model holder, so that each of the participants respectively performs multiple rounds of iterations of iterations of the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the model holder, and uses the K samples to perform back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0158] According to one or more embodiments of the present disclosure, Example 11 provides the method of Example 10, which performs shuffling and copying secret sharing operations of the first data set with the data holder and the model holder so that each of the participants respectively obtains two first shards of the shuffled first data set, including: in response to receiving a fifth shard of the first data set distributed by the data holder in a two-party addition secret sharing manner, shuffling the local fifth shard using a first random seed to obtain a seventh shard; generating a second random data set with the same specification as the seventh shard, and sending the second random data set to the model holder; subtracting the data at corresponding positions of the seventh shard from the second random data set to obtain a fourth data set, determining the fourth data set as the first shard, and sending the fourth data set as the first shard to the data holder; in response to receiving the first random data set sent by the model holder, determining the sum of the first random data set and the second random data set as the first shard.

[0159] According to one or more embodiments of the present disclosure, Example 12 provides an image classification method, which is applied to a model holder, and the method includes: obtaining a first image to be classified; inputting the first image into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of the pre-trained model described in any one of Examples 1-11, and the initial image classification model is a pre-trained model.

[0160] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-12 when executed by a processing device.

[0161] According to one or more embodiments of the present disclosure, Example 14 provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-12.

[0162] According to one or more embodiments of the present disclosure, Example 15 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1-12 when executed by a processor.

[0163] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

[0164] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0165] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A fine-tuning method for a pre-trained model, characterized in that: The participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, and the method is applied to the data holder. The method includes: Training the first model using at least part of the samples in the fine-tuning sample set; For each sample in a first sample set, determining a first sample loss of a first model obtained after training on the sample, wherein the first sample set is a subset of the fine-tuning sample set; Performing a shuffling and copying secret sharing operation of a first data set with the model holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes the first sample set and a first sample loss of each sample in the first sample set; In response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the pre-trained model is fine-tuned for multiple rounds of iterations together with the model holder and the auxiliary computing party based on the two local first shards, so that each of the participating parties respectively performs multiple rounds of iterations of iterations of the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set together with the model holder and the auxiliary computing party, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

2. The method according to claim 1, characterized in that The two first shards of the local second data set, together with the model holder and the auxiliary calculation party, determine K samples with the largest reducible retention loss RHO from the second sample set, including: Based on the two shards of the local second sample set, calculate, with the model holder and the auxiliary computing party, the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set, so that each of the participating parties obtains two third shards of the second sample loss respectively; Calculate the RHO of each sample in the second sample set with the model holder and the auxiliary calculation party according to the two slices of the first sample loss of each sample in the second sample set locally and the two third slices locally, so that each of the participants obtains two fourth slices of the RHO of each sample in the second sample set, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; Based on the two fourth slices of the RHO of each sample in the local second sample set, the model holder and the auxiliary calculation party determine K samples with the largest RHO from the second sample set.

3. The method according to claim 2, characterized in that The two fourth slices based on the RHO of each sample in the local second sample set, together with the model holder and the auxiliary calculation party, determine K samples with the largest RHO from the second sample set, including: Based on the two fourth shards of the RHO of each sample in the local second sample set, the RHO of each sample in the second sample set is sorted by a multi-party secure sorting algorithm with the model holder and the auxiliary computing party, and according to the sorting results, K samples with the largest RHO are determined from the second sample set.

4. The method according to any one of claims 1 to 3, characterized in that The step of performing a shuffling and copying secret sharing operation of the first data set with the model holder and the auxiliary computing party so that each of the participants obtains two first fragments of the first data set obtained after shuffling, comprises: Distribute the first data set to the model holder and the auxiliary computing party in a two-party additive secret sharing manner, so that the model holder and the auxiliary computing party respectively obtain a fifth shard of the first data set, and shuffle the fifth shards held by themselves using a first random seed, generate two first shards of the shuffled first data set based on the shuffled fifth shards, and send the first shard of the two first shards that is not shared by the model holder and the auxiliary computing party to the data holder; The first fragments respectively sent by the model holder and the auxiliary computing party are received as two first fragments of the first data set obtained after shuffling.

5. A method for fine-tuning a pre-trained model, characterized in that: The participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, and the method is applied to the model holder. The method includes: Performing shuffling and copying secret sharing operations of a first data set with the data holder and the auxiliary computing party, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is a sample loss of a first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; Distributing the model parameters of the pre-trained model to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participating parties obtains two second shards of the model parameters respectively; Based on the two first shards of the local data, the pre-trained model is fine-tuned for multiple rounds of iterations together with the data holder and the auxiliary computing party, so that each of the participants respectively performs multiple rounds of iterations of iterations of the two second shards of the local data until the fine-tuning cutoff condition is reached, wherein in each round of iterations ...

6. The method according to claim 5, characterized in that The two first shards of the local second data set, together with the data holder and the auxiliary computing party, determine K samples with the largest reducible retention loss RHO from the second sample set, including: Based on the two shards of the local second sample set, calculate, with the data holder and the auxiliary computing party, the second sample loss of the pre-trained model obtained after the previous round of iterative fine-tuning on each sample in the second sample set, so that each of the participating parties obtains two third shards of the second sample loss respectively; Calculate the RHO of each sample in the second sample set with the data holder and the auxiliary calculation party according to the two shards of the first sample loss of each sample in the second sample set locally and the two third shards locally, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; Based on the two fourth fragments of the RHO of each sample in the local second sample set, determine K samples with the largest RHO from the second sample set together with the data holder and the auxiliary calculation party.

7. The method according to claim 6, characterized in that The two fourth shards of the RHO of each sample in the local second sample set, together with the data holder and the auxiliary calculation party, determine K samples with the largest RHO from the second sample set, including: Based on the two fourth shards of the RHO of each sample in the local second sample set, the RHO of each sample in the second sample set is sorted by a multi-party secure sorting algorithm with the data holder and the auxiliary computing party, and according to the sorting results, K samples with the largest RHO are determined from the second sample set.

8. The method according to claim 5, characterized in that: The step of performing a shuffling and copying secret sharing operation of the first data set with the data holder and the auxiliary computing party so that each of the participants obtains two first fragments of the first data set obtained after shuffling, includes: In response to receiving a fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, shuffling the local fifth shard using a first random seed to obtain a sixth shard; Generate a first random data set with the same specification as the sixth shard, and send the first random data set to the auxiliary computing party; subtracting the data at corresponding positions of the sixth shard from the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holder; In response to receiving a second random data set sent by the auxiliary computing party, a sum of the first random data set and the second random data set is determined as the first shard.

9. The method according to any one of claims 5 to 8, characterized in that: The method further comprises: Acquire the second shard obtained after fine-tuning and not held by the model holder from the data holder or the auxiliary computing party; The second shard obtained from the data holder or the auxiliary computing party is merged with the two local second shards to obtain the model parameters of the pre-trained model after fine-tuning.

10. A method for fine-tuning a pre-trained model, characterized in that: The participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, and the method is applied to the auxiliary computing party. The method includes: Performing a shuffling and copying secret sharing operation of a first data set with the data holder and the model holder, so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is a sample loss of a first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; In response to receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two local first shards, the pre-trained model is fine-tuned for multiple rounds with the data holder and the model holder, so that each of the participants respectively performs multiple rounds of iterative fine-tuning on the two local second shards until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, based on the two first shards of the local second data set, K samples with the largest reducible retention loss RHO from the second sample set with the data holder and the model holder, and uses the K samples for back propagation to update the model parameters, the second sample set is the samples in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

11. The method according to claim 10, characterized in that The performing of the shuffling and copying secret sharing operations of the first data set with the data holder and the model holder so that each of the participants respectively obtains two first fragments of the first data set obtained after shuffling, comprises: In response to receiving a fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, shuffling the local fifth shard using a first random seed to obtain a seventh shard; Generate a second random data set with the same specification as the seventh shard, and send the second random data set to the model holder; subtracting data at corresponding positions of the seventh shard from data of the second random data set to obtain a fourth data set, determining the fourth data set as the first shard, and sending the fourth data set as the first shard to the data holder; In response to receiving a first random data set sent by the model holder, a sum of the first random data set and the second random data set is determined as the first shard.

12. An image classification method, characterized in that: Applied to the model holder, the method includes: Acquire a first image to be classified; The first image is input into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of a pre-trained model according to any one of claims 1-11, and the initial image classification model is a pre-trained model.

13. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 12 are implemented.

14. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images