Pre-training model fine-tuning method, image classification method, medium, device and product

Abstract

A pre-training model fine-tuning method, an image classification method, a medium, an equipment and a product. The method comprises: training a first model; determining sample losses of the first model on each sample after training; performing shuffle and replication secret sharing of a first data set with each other participant, so that each participant obtains two first shards of the first data set after shuffling; receiving two second shards of model parameters of the pre-training model distributed by a model holder, performing multi-round iterative fine-tuning of the pre-training model with each other participant based on the two local first shards, in each round of iterative fine-tuning, the data holder determines K samples with the maximum RHO based on the two first shards of the second data set with each other participant, and updates the model parameters by back propagation using the K samples. Valuable and unlearned samples are selected for model fine-tuning, and the efficiency of safe model fine-tuning is improved. The method can be applied to federated learning.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of cyberspace security, in particular, to a pre-trained model fine-tuning method, an image classification method, a medium, a device and a product. BACKGROUND

[0002] Secure multi-party computation, also known as multi-party secure computation (MPC), can be used to calculate the result of a function by multiple parties without leaking the input data of each party in the function, and the result of the calculation is disclosed to one or more parties. Among them, the secure fine-tuning of the model belongs to a typical application of secure multi-party computation. The secure fine-tuning of the model usually includes two stages: a pre-trained model preparation stage and a secure model update stage using fine-tuning samples. The pre-trained model preparation stage usually uses an open source large dataset to train a general model in plaintext for multiple rounds to obtain a pre-trained model. In the secure model update stage using fine-tuning samples, the fine-tuning samples and the model parameters of the pre-trained model are secret shared, and multiple parties collaboratively perform multiple rounds of iterative fine-tuning to obtain a fine-tuned model. However, the secure fine-tuning scheme of the model has the problem of low efficiency. SUMMARY

[0003] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed technology, nor is it intended to be used to limit the scope of the claimed technology.

[0004] In a first aspect, the disclosure provides a fine-tuning method of a pre-trained model, participants of the fine-tuning of the pre-trained model include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the method is applied to the data holder, and the method comprises: training a first model by using at least part of samples in the fine-tuning sample set; determining, for each sample in a first sample set, a first sample loss of the first model on the sample after training, wherein the first sample set is a subset of the fine-tuning sample set; performing a shuffling and replicated secret sharing operation of a first data set with the model holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set comprises the first sample set and the first sample loss of each sample in the first sample set; in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, performing multi-round iterative fine-tuning of the pre-trained model with the model holder and the auxiliary computing party based on the two first shards locally, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of a second data set locally, K samples with maximum reducible reserved loss RHO from a second sample set with the model holder and the auxiliary computing party, and updates model parameters by using the K samples for back propagation, the second sample set is a sample for the current round of iterative fine-tuning in the first sample set, and the second data set comprises the second sample set and the first sample loss of each sample in the second sample set.

[0005] In a second aspect, the disclosure provides a fine-tuning method of a pre-trained model, participants of the fine-tuning of the pre-trained model include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the method is applied to the model holder, and the method comprises: performing a shuffling and replicated secret sharing operation of a first data set with the data holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first data set obtained after shuffling, wherein the first data set comprises a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is a sample loss of a first model obtained after training on a corresponding sample, and the first model obtained after training is obtained by the data holder training a first model by using at least part of the samples in the fine-tuning sample set; distributing model parameters of the pre-trained model to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants obtains two second shards of the model parameters; based on the two first shards locally, performing multi-round iterative fine-tuning of the pre-trained model with the data holder and the auxiliary computing party, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning stopping condition is reached, wherein in each round of iterative fine-tuning, the model holder determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set, and updates the model parameters by using the K samples for back propagation, the second sample set is a sample in the first sample set used for the current round of iterative fine-tuning, and the second data set comprises the second sample set and a first sample loss of each sample in the second sample set.

[0006] In a third aspect, the disclosure provides a fine-tuning method of a pre-trained model, participants of fine-tuning of the pre-trained model include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the method is applied to the auxiliary computing party, and the method includes: performing a shuffling and replicated secret sharing operation of a first data set with the data holder and the model holder, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is a sample loss of a first model obtained after training on a corresponding sample, and the first model obtained after training is obtained by the data holder training a first model by using at least part of the samples in the fine-tuning sample set; in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, performing multi-round iterative fine-tuning of the pre-trained model with the data holder and the model holder based on the two first shards locally, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set together with the data holder and the model holder, and updates the model parameters by using the K samples for back propagation, the second sample set is a sample for the current round of iterative fine-tuning in the first sample set, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0007] In a fourth aspect, the disclosure provides an image classification method, applied to a model holder, and the method includes: obtaining a first image to be classified; and inputting the first image into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model by the fine-tuning method of the pre-trained model provided in the first aspect to the third aspect of the disclosure, and the initial image classification model is a pre-trained model.

[0008] In a fifth aspect, the disclosure provides a computer readable medium having a computer program stored thereon, the computer program being executed by a processing device to implement the steps of the method provided in the first aspect of the disclosure, or the steps of the method provided in the first aspect of the disclosure, or the steps of the method provided in the third aspect of the disclosure, or the steps of the method provided in the fourth aspect of the disclosure.

[0009] In a sixth aspect, the present disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; and a processing device configured to execute the computer program in the storage device to implement the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the third aspect of the present disclosure, or the steps of the method provided in the fourth aspect of the present disclosure.

[0010] In a seventh aspect, the present disclosure provides a computer program product, comprising a computer program configured to implement the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the first aspect of the present disclosure, or the steps of the method provided in the third aspect of the present disclosure, or the steps of the method provided in the fourth aspect of the present disclosure, when executed by a processor.

[0011] In the technical solution, the data holder first trains the first model by using at least part of the samples in the fine-tuning sample set held by the data holder; then, the first sample loss of the first model on each sample in the first sample set is determined; then, the three participants in the model fine-tuning cooperatively perform the shuffling and replicated secret sharing operation on the first data set, so that each participant obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; meanwhile, the model holder distributes the model parameters of the pre-trained model held by the model holder to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively fine-tune the pre-trained model based on the two first shards locally, so that each participant fine-tunes the two second shards locally for multiple rounds of iteration until the fine-tuning cutoff condition is reached. In each round of iteration, the three participants respectively determine the K samples with the maximum RHO from the second sample set based on the two first shards of the second data set, and update the model parameters by using the K samples, wherein the second sample set is the sample in the first sample set used for the iteration in the current round, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. In the secure fine-tuning of the pre-trained model, the three participants update the model parameters by using the K samples with the maximum RHO in each round of iteration, so that the samples that can be learned, are valuable and have not been learned are selected for model fine-tuning, the waste of calculation on redundant or noise samples is reduced, and thus the number of iterations in the fine-tuning is reduced and the efficiency of the model secure fine-tuning is improved. In addition, the model parameters of the pre-trained model held by the model holder are distributed to the three participants in a replicated secret sharing manner, so that the model parameters are prevented from being leaked. Furthermore, the fine-tuning samples held by the data holder are distributed to the three participants through shuffling and replicated secret sharing, so that the fine-tuning samples are prevented from being leaked, the data holder is prevented from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and the model information of the pre-trained model is prevented from being inferred, and thus the model information is prevented from being leaked. The method can be applied to federated learning.

[0012] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0014] Figure 1 FIG. 4 is a flowchart illustrating a method of fine-tuning a pre-trained model applied to a data holder according to an example embodiment.

[0015] Figure 2 FIG. 5 is a schematic diagram illustrating a fine-tuning process of a pre-trained model according to an example embodiment.

[0016] Figure 3 FIG. 6 is a flowchart illustrating a method of fine-tuning a pre-trained model applied to a model holder according to an example embodiment.

[0017] Figure 4 FIG. 7 is a flowchart illustrating a method of fine-tuning a pre-trained model applied to an auxiliary computing party according to an example embodiment.

[0018] Figure 5 FIG. 8 is a flowchart illustrating an image classification method according to an example embodiment.

[0019] Figure 6 FIG. 9 is a block diagram of a fine-tuning apparatus of a pre-trained model applied to a data holder according to an example embodiment.

[0020] Figure 7 FIG. 10 is a block diagram of a fine-tuning apparatus of a pre-trained model applied to a model holder according to an example embodiment.

[0021] Figure 8 FIG. 11 is a block diagram of a fine-tuning apparatus of a pre-trained model applied to an auxiliary computing party according to an example embodiment.

[0022] Figure 9 FIG. 12 is a block diagram of an image classification apparatus according to an example embodiment.

[0023] Figure 10 FIG. 13 is a structural schematic diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0024] Before introducing specific embodiments of the present disclosure, first, the terms related to the present disclosure are introduced and explained.

[0025] Secret Sharing, also known as secret splitting or secret sharing, is a basic principle of splitting a secret (such as a key, private data, etc.) into multiple shares (also known as shares) and giving them to different data parties for safekeeping. Only when more than a threshold number of parties combine their shares can the secret be restored; no information about the secret can be recovered from the shares obtained from fewer than the threshold number of parties. In multi-party secure computation, the threshold number is usually the same as the number of participants, and the shares into which the secret is split can also be called shares.

[0026] Replicated Secret Sharing (RSS), also known as replicated secret sharing, is an instance of secret sharing used to ensure that a secret value is shared among multiple entities and remains secure in the event of failure or attack of any one of them, involving three or more participants. In RSS, the original secret is split into multiple parts and distributed to multiple entities. Each entity has a partial copy of the original secret, and the original secret value can only be restored when at least a certain number of entities exist simultaneously. In this way, in the event of failure or attack of one of the entities, the other entities can still restore the original secret value through their respective copies.

[0027] For example, in a three-party secure computing scenario using replicated secret sharing, a secret value s can be split into three secret shares (also known as replicated shares) e, g and h, such that s = e + g + h, where the first party holds replicated share e and replicated share g, the second party holds replicated share g and replicated share h, and the third party holds replicated share e and replicated share h. Only when any two of the three parties combine their replicated shares can the secret value s be restored.

[0028] Two-Party Additive Secret Sharing (2PC-Additive) is a secret sharing scheme that allows two participants to add their respective secret shares to restore the original secret value. In two-party additive secret sharing, the original secret value s is split into two shares, which are distributed to two participants. Each participant holds one share of the original secret value, and only when both participants participate can the original secret value be restored through an additive operation.

[0029] For example, in a two-party secure computing scenario using replicated secret sharing, a secret value s can be split into two secret shares (also known as shares) a and b, such that s = a + b, where the first party holds share a and the second party holds share b. Only when both parties combine their shares can the secret value s be restored.

[0030] Shuffling is a technique for protecting data privacy, which allows participants to rearrange data without revealing their own data. The main purpose of shuffling is to ensure the anonymity and irretraceability of data, thereby enhancing privacy protection.

[0031] As discussed in the background, the secure fine-tuning of the model generally includes two stages: a pre-training model preparation stage and a model secure updating stage using fine-tuning samples. The pre-training model preparation stage generally uses an open source large dataset to train a general model under plaintext for multiple rounds to obtain a pre-training model. In the model secure updating stage using fine-tuning samples, the fine-tuning samples and the model parameters of the pre-training model are secret shared, and multiple parties collaboratively perform multiple rounds of iterative fine-tuning to obtain a fine-tuned model. For each round of iterative fine-tuning, all samples of the current round are used to update the model parameters by back propagation, and these samples are not screened, resulting in a large number of iterative fine-tuning times, which affects the efficiency of the secure fine-tuning of the pre-training model.

[0032] In view of this, the present disclosure provides a pre-training model fine-tuning method, an image classification method, a medium, a device and a product.

[0033] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0034] It should be understood that each step described in the method embodiments of the present disclosure can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions of other terms will be given in the description below.

[0036] It should be noted that the "first", "second", and the like concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0038] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0039] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in accordance with relevant laws and regulations.

[0040] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. that performs the operation of the technical solution of the present disclosure according to the prompt information.

[0041] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0042] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation manner of the present disclosure, and other manners that meet the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0043] At the same time, it can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and relevant provisions.

[0044] Figure 1 is a flowchart of a fine-tuning method of a pre-trained model applied to a data holder according to an exemplary embodiment. As shown in Figure 2As shown, the participants in the fine-tuning of the pre-trained model include the data holder, the model holder, and the auxiliary computing party, wherein the data holder holds the fine-tuning sample set, and the model holder holds the pre-trained model. The fine-tuning sample in the fine-tuning sample set includes a single data sample and the annotation result corresponding to the single data sample. The pre-trained model is usually obtained by training the general model in plain text using a large open source data set for multiple rounds. The pre-trained model can be a generation model (for example, a text generation model, an image generation model, etc.), a classification model (for example, a text classification model, an image classification model), a prediction model, and other types of models, which are not specifically limited in this disclosure.

[0045] For example, the pre-trained model is an initial image classification model, and the fine-tuning samples in the fine-tuning sample set include image samples and classification labels corresponding to the image samples.

[0046] like Figure 1 As shown, the fine-tuning method applied to the pre-trained model of the data holder may include S101 to S104.

[0047] In S101 , a first model is trained using at least part of the samples in the fine-tuning sample set.

[0048] In the present disclosure, the first model can be trained using all samples in the fine-tuning sample set, or using a portion of the samples in the fine-tuning sample set. The first model can be any model, for example, a neural network, specifically a residual network (ResNet). When the first model is trained using at least a portion of the samples in the fine-tuning sample set, the cutoff conditions for model training may be that the number of training times reaches a preset number, the model accuracy reaches a first preset accuracy threshold, the model loss is less than a first preset loss threshold, etc.

[0049] In S102 , for each sample in a first sample set, a first sample loss of a first model obtained after training on the sample is determined, wherein the first sample set is a subset of the fine-tuning sample set.

[0050] In the present disclosure, the first model obtained after the above-mentioned S101 training can be used as an irreducible loss (IL) model. Afterwards, for each sample in the first sample set, the sample can be input into the IL model to obtain the predicted value of the IL model. Then, based on the difference between the predicted value and the labeling result corresponding to the sample, the first sample loss of the IL model on the sample is calculated as the IL of the sample.

[0051] In one embodiment, the first sample set is a fine-tuning sample set, that is, the IL of each sample in the fine-tuning sample set is calculated.

[0052] In another implementation, the first sample set is a proper subset of the fine-tuning sample set, for example, the first sample set is the difference set of the fine-tuning sample set and the sample set used for training the first model in S101.

[0053] In S103, the shuffling and replicated secret sharing operation of the first data set is performed by the data holder, the model holder and the auxiliary computing party, so that each participant obtains two first shards of the first data set obtained after shuffling, and the first data set includes the first sample set and the first sample loss of each sample in the first sample set.

[0054] In S104, in response to receiving two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the pre-trained model is fine-tuned by the data holder, the model holder and the auxiliary computing party based on the two local first shards, so that each participant fine-tunes the two local second shards in multiple rounds of iterative fine-tuning until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines K samples with the maximum RHO from the second sample set based on the two first shards of the local second data set, and updates the model parameters using the K samples.

[0055] In the present disclosure, the second sample set is the sample in the first sample set used for the current round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0056] As shown in Figure 2 After obtaining the IL of each sample in the first sample set, the three participants can securely shuffle the first sample set and the IL of each sample in the first sample set (hereinafter referred to as the first data set), and the shuffling and replicated secret sharing operation of the first data set can be performed by the data holder in cooperation with the model holder and the auxiliary computing party, so that each participant obtains two first shards of the first data set obtained after shuffling. The fine-tuning samples held by the data holder are distributed among the three participants after shuffling and replicated secret sharing, which not only avoids the leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards (i.e. the shards of the K samples with the maximum RHO) and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding the leakage of model information.

[0057] In order to avoid the leakage of model parameters (i.e. model weights), the model holder can distribute the model parameters of the pre-trained model to the data holder and the auxiliary computing party in a replicated secret sharing manner (as shown in Figure 2The model holder can distribute the model parameters to the data holder and the auxiliary computing party in a replicated secret sharing manner in the following manner. First, the model holder locally generates a first matrix and a second matrix of the same dimension as the parameter matrix, where the parameter matrix is a matrix composed of the model parameters of the pre-trained model. Then, the model holder subtracts the first matrix and the second matrix from the parameter matrix in turn to obtain a third matrix, i.e., the third matrix = the parameter matrix - the first matrix - the second matrix. After that, the model holder sends the first matrix and the third matrix to the data holder, and the data holder receives the first matrix and the third matrix sent by the model holder and takes them as two second shards of the model parameters. Meanwhile, the model holder sends the second matrix and the third matrix to the auxiliary computing party, and the auxiliary computing party receives the second matrix and the third matrix sent by the model holder and takes them as two second shards of the model parameters. At the same time, the model holder takes the first matrix and the second matrix as two second shards of the model parameters locally. In this way, the model parameters can be distributed in a replicated secret sharing manner among the three participants in the fine-tuning of the pre-trained model.

[0058] After receiving the two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, the data holder can perform multiple rounds of iterative fine-tuning of the pre-trained model in cooperation with the model holder and the auxiliary computing party based on the two first shards of the first data set held by the data holder after shuffling, so that each participant performs multiple rounds of iterative fine-tuning of the two second shards of the model parameters held by itself until the fine-tuning cutoff condition is reached. When performing iterative fine-tuning of the pre-trained model, the first sample set can be input into the pre-trained model in batches to select K samples with the maximum Reducible Holdout Loss (RHO) from the input samples in the current batch according to the output result of the pre-trained model, and then update the model parameters by backpropagation using the respective second sample loss of the selected K samples, to complete one iteration of fine-tuning, where K≥1.

[0059] The meaning of RHO is an estimate of the difference between the loss of the current sample (i.e., the second sample loss) and the loss of the sample when the model is trained well. The larger the RHO, the more the sample needs to be learned. Therefore, K samples with the maximum RHO are selected from the input samples in the current batch for model parameter fine-tuning.

[0060] Specifically, the three participants cooperatively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two first shards of the first data set obtained after shuffling, specifically, the three participants respectively perform multiple rounds of iterative fine-tuning on the two second shards of the model parameters held by themselves, until a fine-tuning stopping condition is reached. The fine-tuning stopping condition can be that the accuracy of the model reaches a second accuracy threshold, the loss of the model is less than a second preset loss threshold, etc.

[0061] In the above technical solution, the data holder first trains the first model using at least part of the samples in the fine-tuning sample set held by itself; then, determines the first sample loss of the first model obtained after training on each sample in the first sample set; then, the three participants of model fine-tuning cooperatively perform shuffling and replicated secret sharing operations on the first data set, so that each participant obtains two first shards of the first data set obtained after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by itself to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively perform multiple rounds of iterative fine-tuning on the pre-trained model based on the two first shards locally, so that each participant performs multiple rounds of iterative fine-tuning on the two second shards locally until a fine-tuning stopping condition is reached. In each round of iterative fine-tuning, the three participants respectively determine the K samples with the largest RHO from the second sample set based on the two first shards of the second data set locally, and update the model parameters by back propagation using the K samples, the second sample set being the samples in the first sample set used for the current round of iterative fine-tuning, and the second data set including the second sample set and the first sample loss of each sample in the second sample set. When the three participants perform secure fine-tuning on the pre-trained model, the K samples with the largest RHO are used for model parameter update in each round of iterative fine-tuning, so that samples that can be learned, are valuable and have not been learned are selected for model fine-tuning, reducing the waste of calculation on redundant or noisy samples, thereby reducing the number of iterations of fine-tuning and improving the efficiency of model secure fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid leakage of model parameters. Furthermore, the fine-tuning samples held by the data holder are distributed among the three participants after shuffling and replicated secret sharing, which not only avoids leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding leakage of model information. This method can be applied to federated learning.

[0062] The following detailed description of the implementation of the data holder, the model holder and the auxiliary computing party performing the shuffling and replication secret sharing operation of the first data set in S103 above will be described in detail. Specifically, it can be achieved by the following steps (a1) and (a2).

[0063] Step (a1): The data holder can distribute the first data set to the model holder and the auxiliary computing party in a two-party additive secret sharing manner, so that the model holder and the auxiliary computing party respectively obtain a fifth shard of the first data set, and respectively shuffle the fifth shard held by themselves using the first random seed, generate two first shards of the first data set obtained after shuffling based on the fifth shard obtained after shuffling, and send the first shard not common to the model holder and the auxiliary computing party to the data holder.

[0064] Step (a2): The data holder receives the first shard sent by the model holder and the auxiliary computing party respectively as the two first shards of the first data set obtained after shuffling.

[0065] Specifically, the data holder can distribute the first data set to the model holder and the auxiliary computing party in a two-party additive secret sharing manner in the following way:

[0066] First, the data holder can generate a third random data set with the same specification as the first data set locally; then, the data holder subtracts the data at the corresponding positions of the first data set and the third random data set to obtain a fifth data set; finally, the data holder sends the third random data set and the fifth data set as two shards of the first data set to the model holder and the auxiliary computing party, respectively, for example, the data holder sends the third random data set to the model holder, and the model holder takes the third random data set as a shard (referred to as a fifth shard) of the first data set after receiving it, and the data holder sends the fifth data set to the auxiliary computing party, and the auxiliary computing party takes the fifth data set as another shard (referred to as a fifth shard) of the first data.

[0067] Among them, the same specification of the two data sets means that the two data sets are consistent in structure and format. Specifically, it can include the following aspects:

[0068] The dimensions of the data set: the two data sets have the same number of rows and columns;

[0069] Data type: the data at the corresponding positions has the same data type, such as integer, floating point number or string, etc.

[0070] Data structure: the data in both datasets is organized in the same way, for example, both are in table form;

[0071] Data range: the data values in both datasets can come from the same value range, for example, both are integers between 0 and 100.

[0072] The third random dataset is the same specification as the first dataset, which means that the dimensions, data types, data structures, and data ranges of the third random dataset are the same as those of the first dataset.

[0073] Specifically, the model holder and the auxiliary computing party can realize the shuffling and replicated secret sharing of the first dataset in the following way:

[0074] After receiving a fifth shard of the first dataset distributed by the data holder in a two-party additive secret sharing manner, the model holder can shuffle the fifth shard using the first random seed to obtain a sixth shard. Then, the model holder generates a first random dataset with the same specification as the sixth shard and sends it to the auxiliary computing party. Next, the model holder subtracts the data at the corresponding positions of the sixth shard and the first random dataset to obtain a third dataset, determines the third dataset as a shard of the first dataset after shuffling, and sends the third dataset as a shard of the first dataset after shuffling to the data holder. The first random dataset has the same specification as the sixth shard, which means that the dimensions, data types, data structures, and data ranges of the first random dataset are the same as those of the sixth shard, and in essence, the same as those of the first dataset.

[0075] At the same time, after receiving a fifth shard of the first dataset distributed by the data holder in a two-party additive secret sharing manner, the auxiliary computing party can shuffle the fifth shard using the first random seed to obtain a seventh shard. Then, the auxiliary computing party generates a second random dataset with the same specification as the seventh shard and sends it to the model holder. Next, the auxiliary computing party subtracts the data at the corresponding positions of the seventh shard and the second random dataset to obtain a fourth dataset, determines the fourth dataset as a shard of the first dataset after shuffling, and sends the fourth dataset as a shard of the first dataset after shuffling to the data holder. The second random dataset has the same specification as the seventh shard, which means that the dimensions, data types, data structures, and data ranges of the second random dataset are the same as those of the seventh shard, and in essence, the same as those of the first dataset.

[0076] After receiving the second random data set sent by the auxiliary computing party, the model holder can determine the sum of the locally generated first random data set and the second random data set received from the auxiliary computing party as another shard of the shuffled first data set. That is, after the shuffling and replicated secret sharing operations of the first data set, the model holder can hold two shards of the shuffled first data set, which are the third data set and the sum of the first random data set and the second random data set, respectively.

[0077] After receiving the first random data set sent by the model holder, the auxiliary computing party can determine the sum of the first random data set received from the model holder and the locally generated second random data set as another shard of the shuffled first data set. That is, after the shuffling and replicated secret sharing operations of the first data set, the auxiliary computing party can hold two shards of the shuffled first data set, which are the fourth data set and the sum of the first random data set and the second random data set, respectively.

[0078] After receiving the third data set sent by the model holder and the fourth data set sent by the auxiliary computing party, the data holder takes them as two first shards of the shuffled first data set. After the shuffling and replicated secret sharing operations of the first data set, the data holder can hold two shards of the shuffled first data set, which are the third data set and the fourth data set, respectively.

[0079] The first random seed is a random seed shared by the auxiliary computing party and the model holder. The model holder and the auxiliary computing party use the same random seed (i.e., the first random seed) to shuffle the shards of the first data set held by each of them. This can ensure that the data holder and the auxiliary computing party disrupt the order of the shards of the data set held by themselves consistently, thereby ensuring the accuracy of subsequent model fine-tuning calculations. It should be noted that the specific way of shuffling the data set using the random seed is known to those skilled in the art, and the present disclosure will not be repeated here.

[0080] The following describes the specific implementation of the data holder in S104 based on the two first shards of the local second data set and the K samples with the maximum reduced reserved loss RHO determined by the model holder and the auxiliary computing party from the second sample set. Specifically, the following steps (b1) to (b3) can be used to achieve this.

[0081] Step (b1): The data holder calculates, based on the two shards of the local second sample set, the second sample loss of each sample in the second sample set of the pre-trained model obtained after the last round of iterative fine-tuning, together with the model holder and the auxiliary computing party, so that each participant obtains two third shards of the second sample loss, respectively.

[0082] In the present disclosure, the three participants securely compute the second sample loss of the pre-trained model on each sample in the second sample set after the fine-tuning of the last iteration based on the two shards of the local second sample set. Specifically, as shown in Figure 2 The three participants can perform secure forward propagation on the samples for the fine-tuning of the current iteration, i.e., the second sample set, by means of the secure addition protocol, the secure multiplication protocol, the secure comparison protocol, and the truncation protocol, to compute the second sample loss of the pre-trained model on each sample in the second sample set after the fine-tuning of the last iteration, so that the three participants obtain two third shards of the respective second sample loss of each sample in the second sample set.

[0083] Step (b2): The data holder computes the respective RHO of each sample in the second sample set based on the two shards of the respective first sample loss of each sample in the local second sample set and the two third shards, together with the model holder and the auxiliary computing party, so that each participant obtains two fourth shards of the respective RHO of each sample in the second sample set.

[0084] In the present disclosure, the RHO of a sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample. After each of the three participants obtains two third shards of the respective second sample loss of each sample in the second sample set, the three participants can securely compute the respective RHO of each sample in the second sample set based on the two shards of the respective first sample loss of each sample in the second sample set held by the participant and the two third shards of the respective second sample loss of each sample in the second sample set, so that each participant obtains two fourth shards of the respective RHO of each sample in the second sample set.

[0085] Specifically, for each sample in the second sample set, the three participants can securely compute the respective RHO of each sample in the second sample set based on the two shards of the first sample loss of the sample held by the participant and the two third shards of the second sample loss of the sample, so that each participant obtains two fourth shards of the RHO of the sample.

[0086] Step (b3): The data holder determines the K samples with the largest RHO from the second sample set based on the two fourth shards of the respective RHO of each sample in the local second sample set, together with the model holder and the auxiliary computing party.

[0087] In a possible implementation, the data holder can sort the RHOs of the samples in the second sample set based on the two fourth shards of the respective RHO of each sample in the second sample set, and sort the RHOs of the samples in the second sample set by the multi-party secure sorting algorithm with the model holder and the auxiliary computing party, and determine the K samples with the largest RHO from the second sample set according to the sorting result. It should be noted that the specific implementation of the multi-party secure sorting algorithm is known to those skilled in the art, and the present disclosure will not be repeated here.

[0088] For example, as shown in Figure 2 The second sample set includes sample x2, sample x0, and sample x1, which are input into the pre-trained model obtained after the last round of iterative fine-tuning to perform secure forward propagation and calculate the second sample loss of the pre-trained model obtained after the last round of iterative fine-tuning on sample x2, sample x0, and sample x1, respectively, to obtain the second sample loss loss2 of sample x2, the second sample loss loss0 of sample x0, and the second sample loss loss1 of sample x1. The first sample loss of each sample in the second sample set obtained by S102 is as follows: the first sample loss of sample x2 is IL loss2, the first sample loss of sample x0 is IL loss0, and the first sample loss of sample x1 is IL loss1. Thus, the RHO of each sample can be calculated, i.e., the RHO of sample x2 is RHO loss2, the RHO of sample x0 is RHO loss0, and the RHO of sample x1 is RHO loss1. The sorting result obtained by sorting these RHOs by the multi-party secure sorting algorithm is: RHO loss2 (the sequence number of the sorting result is 0), RHO loss1 (the sequence number of the sorting result is 1), and RHO loss0. If K = 2, the K samples with the largest RHO include sample x2 corresponding to RHO loss2 and sample x1 corresponding to RHO loss1. Finally, the second sample loss loss2 of sample x2 and the second sample loss loss2 of sample x1 are used to update the model parameters by back propagation.

[0089] Figure 3 is a flowchart of a pre-trained model fine-tuning method applied to a model holder according to an example embodiment. The participants in the pre-trained model fine-tuning include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds a pre-trained model. As shown in Figure 3 The pre-trained model fine-tuning method applied to the model holder can include the following S201-S203.

[0090] In S201, a shuffling and replicated secret sharing operation of a first data set is performed with the data holder and the auxiliary computing party, so that each participant obtains two first shards of the first data set obtained after shuffling, the first data set including a first sample set and a first sample loss of each sample in the first sample set, the first sample set being a subset of the fine-tuning sample set.

[0091] wherein the first sample loss is a sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder training the first model using at least part of the samples in the fine-tuning sample set.

[0092] In S202, the model parameters of the pre-trained model are distributed to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters.

[0093] In S203, based on the two first shards locally, the pre-trained model is fine-tuned with the data holder and the auxiliary computing party in multiple rounds of iterative fine-tuning, so that each participant performs multiple rounds of iterative fine-tuning on the two second shards locally until the fine-tuning stopping condition is reached, wherein in each round of iterative fine-tuning, the model holder determines K samples with the maximum reducible reserved loss RHO from the second sample set based on the two first shards of the second data set locally, and updates the model parameters using the K samples.

[0094] wherein the second sample set is a sample in the first sample set used for the current round of iterative fine-tuning, and the second data set includes the second sample set and a first sample loss of each sample in the second sample set.

[0095] In the technical solution, the data holder first trains the first model using at least part of the fine-tuning sample set held by the data holder; then, determines the first sample loss of the first model on each sample in the first sample set; then, the three participants of model fine-tuning cooperatively perform shuffling and replicated secret sharing of the first data set, so that each participant obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by the model holder to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively fine-tune the pre-trained model based on the two first shards locally, so that each participant fine-tunes the two second shards locally for multiple rounds of iteration until the fine-tuning cutoff condition is reached. In each round of iteration, the three participants respectively determine the K samples with the maximum RHO from the second sample set based on the two first shards of the second data set locally, and update the model parameters by back propagation using the K samples, wherein the second sample set is the sample in the first sample set used for the current round of iteration, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants securely fine-tune the pre-trained model, the K samples with the maximum RHO are used for model parameter update in each round of iteration, so that the samples that can be learned, valuable and not yet learned are selected for model fine-tuning, reducing the waste of calculation on redundant or noisy samples, thereby reducing the number of iterations of fine-tuning and improving the efficiency of model fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid leakage of model parameters. Furthermore, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and replicated secret sharing, which not only avoids leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding leakage of model information. The method can be applied to federated learning.

[0096] Optionally, the model holder determines, based on the two first shards of the local second data set, the K samples with the maximum reducible retention loss RHO from the second sample set together with the data holder and the auxiliary computing party, including: the model holder calculates, based on the two shards of the local second sample set, the second sample loss of each sample in the second sample set on the pre-trained model obtained after fine-tuning in the last iteration together with the data holder and the auxiliary computing party, so that each participant obtains two third shards of the second sample loss respectively; the model holder calculates, according to the two shards of the first sample loss of each sample in the local second sample set and the two local third shards, the RHO of each sample in the second sample set together with the data holder and the auxiliary computing party, so that each participant obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of a sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; and the model holder determines, based on the two fourth shards of the RHO of each sample in the local second sample set, the K samples with the maximum RHO from the second sample set together with the data holder and the auxiliary computing party.

[0097] Optionally, the model holder determines, based on the two fourth shards of the RHO of each sample in the local second sample set, the K samples with the maximum RHO from the second sample set together with the data holder and the auxiliary computing party, including: the model holder sorts, based on the two fourth shards of the RHO of each sample in the local second sample set, the RHO of each sample in the second sample set together with the data holder and the auxiliary computing party by using a multi-party secure sorting algorithm, and determines the K samples with the maximum RHO from the second sample set according to the sorting result.

[0098] Optionally, the S201 can include: in response to receiving one fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, shuffling the local fifth shard by using a first random seed to obtain a sixth shard; generating a first random data set with the same specification as the sixth shard, and sending the first random data set to the auxiliary computing party; subtracting the data at the corresponding positions of the sixth shard and the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holder; and in response to receiving the second random data set sent by the auxiliary computing party, determining the sum of the first random data set and the second random data set as the first shard.

[0099] In addition, as Figure 2As shown, after completing the fine-tuning of the pre-trained model, the three participants can respectively obtain two second shards of model parameters of the fine-tuned pre-trained model, and the model holder can obtain the second shard of the fine-tuned pre-trained model not held by the model holder from the data holder or the auxiliary computing party, and merge the three shards of the fine-tuned pre-trained model to obtain the model parameters of the fine-tuned pre-trained model, so as to restore the fine-tuned pre-trained model. Specifically, the fine-tuning method of the pre-trained model applied to the model holder can further include the following two steps: obtaining the second shard of the fine-tuned pre-trained model not held by the model holder from the data holder or the auxiliary computing party; and merging the second shard obtained from the data holder or the auxiliary computing party with the two second shards locally to obtain the model parameters of the fine-tuned pre-trained model.

[0100] The specific implementation manners of the steps in the fine-tuning method of the pre-trained model applied to the model holder according to the embodiments of the present disclosure have been described in detail in the fine-tuning method of the pre-trained model applied to the data holder according to the embodiments of the present disclosure, and will not be repeated here.

[0101] Figure 4 is a flowchart of a fine-tuning method of a pre-trained model applied to an auxiliary computing party according to an exemplary embodiment. Wherein, the participants of the pre-trained model fine-tuning include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, and the model holder holds a pre-trained model. As shown in Figure 4 The fine-tuning method of the pre-trained model applied to the auxiliary computing party can include the following S301 and S302.

[0102] In S301, the shuffling and replicated secret sharing operation of the first data set is performed with the data holder and the model holder, so that each participant obtains two first shards of the shuffled first data set, and the first data set includes a first sample set and a first sample loss of each sample in the first sample set, and the first sample set is a subset of the fine-tuning sample set.

[0103] Wherein, the first sample loss is the sample loss of the trained first model on the corresponding sample, and the trained first model is obtained by training the first model with at least part of the samples in the fine-tuning sample set by the data holder.

[0104] In S302, in response to receiving two second shards of model parameters of a pre-trained model distributed by a model holder in a secret sharing manner, based on two first shards locally, the pre-trained model is iteratively fine-tuned with the data holder and the model holder in multiple rounds until a fine-tuning stop condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines, based on the two first shards of the second data set locally, the K samples with the maximum reducible reserved loss RHO from the second sample set together with the data holder and the model holder, and updates the model parameters by back propagation using the K samples.

[0105] wherein the second sample set is the samples in the first sample set used for the current round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0106] In the technical solution, the data holder first trains the first model using at least part of the fine-tuning sample set held by the data holder; then, determines the first sample loss of the first model on each sample in the first sample set; then, the three participants of model fine-tuning cooperatively perform shuffling and replicated secret sharing of the first data set, so that each participant obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by the model holder to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively fine-tune the pre-trained model based on the two first shards locally, so that each participant fine-tunes the two second shards locally for multiple rounds of iteration until the fine-tuning cutoff condition is reached. In each round of iteration, the three participants respectively determine the K samples with the maximum RHO from the second sample set based on the two first shards of the second data set locally, and update the model parameters by back propagation using the K samples, wherein the second sample set is the sample in the first sample set used for the current round of iteration, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants securely fine-tune the pre-trained model, the K samples with the maximum RHO are used for model parameter update in each round of iteration, so that the samples that can be learned, valuable and not yet learned are selected for model fine-tuning, reducing the waste of calculation on redundant or noisy samples, thereby reducing the number of iterations of fine-tuning and improving the efficiency of model fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid leakage of model parameters. Furthermore, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and replicated secret sharing, which not only avoids leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding leakage of model information. The method can be applied to federated learning.

[0107] Optionally, the auxiliary computing party determines, based on the two first shards of the local second sample set, the K samples with the maximum RHO from the second sample set together with the data holder and the model holder, comprising: the auxiliary computing party calculates, based on the two shards of the local second sample set, the second sample loss of each sample in the second sample set on the pre-trained model obtained after fine-tuning in the last iteration together with the data holder and the model holder, so that each participant obtains two third shards of the second sample loss respectively; the auxiliary computing party calculates, according to the two shards of the first sample loss of each sample in the local second sample set and the two third shards, the RHO of each sample in the second sample set together with the data holder and the model holder, so that each participant obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of a sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; and the auxiliary computing party determines, based on the two fourth shards of the RHO of each sample in the local second sample set, the K samples with the maximum RHO from the second sample set together with the data holder and the model holder.

[0108] Optionally, the auxiliary computing party determines, based on the two fourth shards of the RHO of each sample in the local second sample set, the K samples with the maximum RHO from the second sample set together with the data holder and the model holder, comprising: the auxiliary computing party sorts the RHO of each sample in the second sample set by a multi-party secure sorting algorithm based on the two fourth shards of the RHO of each sample in the local second sample set together with the data holder and the model holder, and determines the K samples with the maximum RHO from the second sample set according to the sorting result.

[0109] Optionally, the S301 can comprise: in response to receiving one fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, shuffling the local fifth shard by using a first random seed to obtain a seventh shard; generating a second random data set with the same specification as the seventh shard, and sending the second random data set to the model holder; subtracting the data at the corresponding positions of the seventh shard and the second random data set to obtain a fourth data set, determining the fourth data set as the first shard, and sending the fourth data set as the first shard to the data holder; and in response to receiving the first random data set sent by the model holder, determining the sum of the first random data set and the second random data set as the first shard.

[0110] The specific implementation manners of the steps in the method for fine-tuning the pre-trained model of the auxiliary computing party according to the embodiments of the present disclosure have been described in detail in the method for fine-tuning the pre-trained model of the data holder according to the embodiments of the present disclosure, and will not be repeated here.

[0111] Figure 5is a flow chart of an image classification method according to an exemplary embodiment. Wherein the image classification method can be applied to a model holder, such as Figure 5 As shown, the image classification method can include the following S401 and S402.

[0112] In S401, a first image to be classified is obtained.

[0113] In the present disclosure, the first image can be any type of image, for example, a medical image, which is not limited in the present disclosure.

[0114] In S402, the first image is input into a target image classification model to obtain a classification result of the first image.

[0115] In the present disclosure, the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of the pre-trained model provided by the present disclosure, wherein the initial image classification model is a pre-trained model.

[0116] By fine-tuning the initial image classification model according to the fine-tuning method of the pre-trained model provided by the present disclosure, the fine-tuning efficiency of the initial image classification model can be improved.

[0117] Figure 6 is a block diagram of a pre-trained model fine-tuning device applied to a data holder according to an exemplary embodiment. Wherein the participants of the pre-trained model fine-tuning include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, such as Figure 6As shown, the fine-tuning device 500 applied to the pre-trained model of the data holder includes: a training module 501, configured to train the first model by using at least part of the samples in the fine-tuning sample set; a determination module 502, configured to determine, for each sample in the first sample set, a first sample loss of the first model on the sample after training, wherein the first sample set is a subset of the fine-tuning sample set; a first secure shuffle module 503, configured to perform a shuffle and replicated secret sharing operation of the first data set with the model holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; a first secure fine-tuning module 504, configured to, in response to receiving two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, perform multi-round iterative fine-tuning on the pre-trained model with the model holder and the auxiliary computing party based on the two first shards locally, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning stopping condition is reached, wherein in each round of iterative fine-tuning, the data holder determines K samples with the maximum reducible reserved loss RHO from the second sample set based on the two first shards of the second data set locally with the model holder and the auxiliary computing party, and updates the model parameters by using the K samples, the second sample set is the sample in the first sample set used for the current round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0118] In the technical solution, the data holder first trains the first model using at least part of the fine-tuning sample set held by the data holder; then, determines the first sample loss of the first model on each sample in the first sample set; then, the three participants of model fine-tuning cooperatively perform shuffling and replicated secret sharing of the first data set, so that each participant obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by the model holder to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively fine-tune the pre-trained model based on the two first shards locally, so that each participant fine-tunes the two second shards locally for multiple rounds of iteration until the fine-tuning cutoff condition is reached. In each round of iteration, the three participants respectively determine the K samples with the maximum RHO from the second sample set based on the two first shards of the second data set locally, and update the model parameters by back propagation using the K samples, wherein the second sample set is the sample in the first sample set used for the current round of iteration, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants securely fine-tune the pre-trained model, the K samples with the maximum RHO are used for model parameter update in each round of iteration, so that the samples that can be learned, valuable and not yet learned are selected for model fine-tuning, reducing the waste of calculation on redundant or noisy samples, thereby reducing the number of iterations of fine-tuning and improving the efficiency of model fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid leakage of model parameters. Furthermore, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and replicated secret sharing, which not only avoids leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding leakage of model information. The method can be applied to federated learning.

[0119] Optionally, the first secure fine-tuning module 504 comprises: a first calculation submodule, configured to calculate, based on two shards of a local second sample set, a second sample loss of a pre-trained model obtained after a last round of iteration fine-tuning by the model holder and the auxiliary calculation party on each sample in the second sample set, so that each of the parties obtains two third shards of the second sample loss respectively; a second calculation submodule, configured to calculate, according to the two shards of the first sample loss of each sample in the local second sample set and the two third shards, a RHO of each sample in the second sample set by the model holder and the auxiliary calculation party, so that each of the parties obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; and a first determination submodule, configured to determine, based on the two fourth shards of the RHO of each sample in the local second sample set, K samples with the largest RHO from the second sample set by the model holder and the auxiliary calculation party.

[0120] Optionally, the first determination submodule is configured to sort, based on the two fourth shards of the RHO of each sample in the local second sample set, the RHO of each sample in the second sample set by the model holder and the auxiliary calculation party through a multi-party secure sorting algorithm, and determine, according to a sorting result, K samples with the largest RHO from the second sample set.

[0121] Optionally, the first secure shuffling module 503 comprises: a secret sharing submodule, configured to distribute the first data set to the model holder and the auxiliary calculation party in a two-party additive secret sharing manner, so that the model holder and the auxiliary calculation party obtain one fifth shard of the first data set respectively, and generate two first shards of the first data set after shuffling based on the fifth shard held by each party after shuffling based on a first random seed, and send a first shard that is not common to the model holder and the auxiliary calculation party to the data holder; and a receiving submodule, configured to receive the first shard sent by the model holder and the auxiliary calculation party respectively as the two first shards of the first data set after shuffling.

[0122] Figure 7 FIG. 1 is a block diagram of a pre-trained model fine-tuning device applied to a model holder according to an example embodiment. The pre-trained model fine-tuning device comprises a secure shuffling module 501, a secure fine-tuning module 504, a secure sorting module 505, and a secure determination module 506. Figure 7As shown, the fine-tuning device 600 applied to the pre-trained model of the model holder includes: a second secure shuffle module 601, configured to perform a shuffle and replication secret sharing operation of a first data set with the data holder and the auxiliary computing party, so that each of the participants obtains two first shards of the shuffled first data set respectively, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a trained first model on a corresponding sample, and the trained first model is obtained by training a first model by the data holder using at least part of the samples in the fine-tuning sample set; a model parameter replication secret sharing module 602, configured to distribute model parameters of the pre-trained model to the data holder and the auxiliary computing party in a replication secret sharing manner, so that each of the participants obtains two second shards of the model parameters respectively; and a second secure fine-tuning module 603, configured to perform multi-round iterative fine-tuning of the pre-trained model based on the two first shards locally with the data holder and the auxiliary computing party, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning stop condition is reached, wherein in each round of iterative fine-tuning, the model holder determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set locally with the data holder and the auxiliary computing party, and updates the model parameters by back propagation using the K samples, and the second sample set is a sample for the current round of iterative fine-tuning in the first sample set, and the second data set includes the second sample set and a first sample loss of each sample in the second sample set.

[0123] In the technical solution, the data holder first trains the first model using at least part of the fine-tuning sample set held by the data holder; then, determines the first sample loss of the first model on each sample in the first sample set; then, the three participants of model fine-tuning cooperatively perform shuffling and replicated secret sharing of the first data set, so that each participant obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by the model holder to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively fine-tune the pre-trained model based on the two first shards locally, so that each participant fine-tunes the two second shards locally for multiple rounds of iteration until the fine-tuning cutoff condition is reached. In each round of iteration, the three participants respectively determine the K samples with the maximum RHO from the second sample set based on the two first shards of the second data set locally, and update the model parameters by back propagation using the K samples, wherein the second sample set is the sample in the first sample set used for the current round of iteration, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants securely fine-tune the pre-trained model, the K samples with the maximum RHO are used for model parameter update in each round of iteration, so that the samples that can be learned, valuable and not yet learned are selected for model fine-tuning, reducing the waste of calculation on redundant or noisy samples, thereby reducing the number of iterations of fine-tuning and improving the efficiency of model fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid leakage of model parameters. Furthermore, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and replicated secret sharing, which not only avoids leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding leakage of model information. The method can be applied to federated learning.

[0124] Optionally, the second secure fine-tuning module 603 comprises: a third calculation submodule for calculating, based on two shards of a local second sample set, a second sample loss of a pre-training model obtained after a previous round of iteration fine-tuning on each sample in the second sample set with the data holder and the auxiliary computing party, so that each of the participants obtains two third shards of the second sample loss respectively; a third calculation submodule for calculating, according to the two shards of the first sample loss of each sample in the local second sample set and the two third shards, a RHO of each sample in the second sample set with the data holder and the auxiliary computing party, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; and a second determination submodule for determining, based on the two fourth shards of the RHO of each sample in the local second sample set, K samples with the largest RHO from the second sample set with the data holder and the auxiliary computing party.

[0125] Optionally, the second determination submodule is configured to sort the RHO of each sample in the second sample set by a multi-party secure sorting algorithm based on the two fourth shards of the RHO of each sample in the local second sample set with the data holder and the auxiliary computing party, and determine K samples with the largest RHO from the second sample set according to the sorting result.

[0126] Optionally, the second secure shuffling module 601 comprises: a first shuffling submodule for shuffling, in response to receiving a fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, the local fifth shard by using a first random seed to obtain a sixth shard; a first generation submodule for generating a first random data set with the same specification as the sixth shard, and sending the first random data set to the auxiliary computing party; a third determination submodule for subtracting data at corresponding positions of the sixth shard and the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holder; and a fourth determination submodule for determining, in response to receiving a second random data set sent by the auxiliary computing party, a sum of the first random data set and the second random data set as the first shard.

[0127] Optionally, the fine-tuning device 600 applied to the pre-trained model of the model holder also includes: a first acquisition module, used to obtain the second slice obtained after fine-tuning and not held by the model holder from the data holder or the auxiliary computing party; a merging module, used to merge the second slice obtained from the data holder or the auxiliary computing party with the two local second slices to obtain the model parameters of the pre-trained model obtained after fine-tuning.

[0128] Figure 8 This is a block diagram of a fine-tuning device for a pre-trained model applied to an auxiliary computing party according to an exemplary embodiment. Participants in the fine-tuning of the pre-trained model include a data holder, a model holder, and an auxiliary computing party. The data holder holds a fine-tuning sample set, and the model holder holds the pre-trained model. Figure 8 As shown, the fine-tuning device 700 for the pre-trained model applied to the auxiliary computing party includes: a third secure shuffling module 701, which is used to perform shuffling and copy secret sharing operations on the first data set with the data holder and the model holder, so that each of the participating parties respectively obtains two first fragments of the first data set obtained after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is the sample loss of the first model obtained after training on the corresponding sample, and the first model obtained after training is obtained by the data holder using at least part of the samples in the fine-tuning sample set to train the first model; a third secure fine-tuning module 702, which is used to respond to the receipt of the data holder's request for the first model to be shuffled and copied secret sharing. The two second shards of the model parameters of the distributed pre-trained model are based on the two local first shards, and the pre-trained model is fine-tuned for multiple rounds with the data holder and the model holder, so that each of the participants performs multiple rounds of iterative fine-tuning on the two local second shards respectively until the fine-tuning cutoff condition is reached. In each round of iterative fine-tuning, the auxiliary computing party determines, with the data holder and the model holder, the K samples with the largest reducible retention loss RHO from the second sample set based on the two first shards of the local second data set, and uses the K samples for back propagation to update the model parameters. The second sample set is the sample in the first sample set used for this round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0129] In the technical solution, the data holder first trains the first model using at least part of the fine-tuning sample set held by the data holder; then, determines the first sample loss of the first model on each sample in the first sample set; then, the three participants of model fine-tuning cooperatively perform the shuffling and replicated secret sharing operation of the first data set, so that each participant obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; at the same time, the model holder distributes the model parameters of the pre-trained model held by the model holder to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each participant obtains two second shards of the model parameters; finally, the three participants respectively fine-tune the pre-trained model based on the two first shards locally, so that each participant fine-tunes the two second shards locally for multiple rounds of iteration until the fine-tuning cutoff condition is reached. In each round of iteration, the three participants respectively determine the K samples with the maximum RHO from the second sample set based on the two first shards of the second data set locally, and update the model parameters by back propagation using the K samples, and the second sample set is the sample in the first sample set used for the iteration of the current round, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set. When the three participants securely fine-tune the pre-trained model, the K samples with the maximum RHO are used for model parameter update in each round of iteration, so that the samples that can be learned, valuable and not yet learned can be selected for model fine-tuning, reducing the waste of calculation on redundant or noisy samples, thereby reducing the number of iterations of fine-tuning and improving the efficiency of model fine-tuning. In addition, the model parameters of the pre-trained model held by the model holder are distributed among the three participants in a replicated secret sharing manner, which can avoid leakage of model parameters. Furthermore, the fine-tuning samples held by the data holder are distributed among the three participants through shuffling and replicated secret sharing, which not only avoids leakage of fine-tuning samples, but also avoids the data holder from inferring the selected samples based on the correspondence between the selected sample shards and the fine-tuning samples, and further inferring the model information of the pre-trained model, thereby avoiding leakage of model information. The method can be applied to federated learning.

[0130] Optionally, the third secure shuffle module 701 comprises: a second shuffle submodule, configured to shuffle a fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner using a first random seed to obtain a seventh shard; a second generation submodule, configured to generate a second random data set with the same specification as the seventh shard, and send the second random data set to the model holder; a fifth determination submodule, configured to subtract the seventh shard from data at a corresponding position of the second random data set to obtain a fourth data set, determine the fourth data set as the first shard, and send the fourth data set as the first shard to the data holder; and a sixth determination submodule, configured to determine a sum of the first random data set and the second random data set as the first shard in response to receiving the first random data set sent by the model holder.

[0131] Figure 9 is a block diagram of an image classification device according to an exemplary embodiment. Wherein the image classification device is applied to a model holder, such as Figure 9 As shown, the image classification device 800 comprises: a second acquisition module 801 configured to acquire a first image to be classified; and a classification module 802 configured to input the first image into an image classification model to obtain a classification result of the first image, wherein the first image is input into a target image classification model to obtain the classification result of the first image, and the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of the pre-trained model provided in the present disclosure.

[0132] By fine-tuning the initial image classification model according to the fine-tuning method of the pre-trained model provided in the present disclosure, the fine-tuning efficiency of the initial image classification model can be improved.

[0133] Reference is made below to Figure 10 which shows a structural schematic diagram of an electronic device (such as a terminal device or a server) 900 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0134] As Figure 10As shown, the electronic device 900 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage device 908. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0135] Generally, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication devices 909 can allow the electronic device 900 to communicate wirelessly or wired with other devices to exchange data. Although Figure 10 The electronic device 900 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0136] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 909, or installed from the storage devices 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0137] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer the program for use by or in connection with the instruction execution system, apparatus or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0138] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0139] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and can be accessed via the electronic device.

[0140] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: train the first model using at least part of the fine-tuning sample set, the participants of the pre-trained model fine-tuning including a data holder, a model holder and an auxiliary computing party, the data holder holding the fine-tuning sample set, the model holder holding the pre-trained model; determine, for each sample in the first sample set, a first sample loss of the first model on the sample after training, wherein the first sample set is a subset of the fine-tuning sample set; perform a shuffling and replicated secret sharing operation of the first data set with the model holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set includes the first sample set and the first sample loss of each sample in the first sample set; in response to receiving two second shards of the model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two first shards locally, perform multi-round iterative fine-tuning of the pre-trained model with the model holder and the auxiliary computing party, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, based on the two first shards of the second data set locally, the K samples with the maximum reducible reserved loss RHO from the second sample set with the model holder and the auxiliary computing party, and updates the model parameters using the K samples, the second sample set being the samples in the first sample set used for the current round of iterative fine-tuning, and the second data set including the second sample set and the first sample loss of each sample in the second sample set.

[0141] Or, the computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: perform a shuffling and replicated secret sharing operation of a first data set with a data holder and an auxiliary computing party, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the participants of the pre-trained model fine-tuning include the data holder, a model holder and the auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a first model obtained after training on the corresponding sample, the first model obtained after training is obtained by the data holder training the first model using at least part of the samples in the fine-tuning sample set; distribute the model parameters of the pre-trained model to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants obtains two second shards of the model parameters; based on the two first shards locally, perform multi-round iterative fine-tuning of the pre-trained model with the data holder and the auxiliary computing party, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until the fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set with the data holder and the auxiliary computing party, and updates the model parameters using the K samples, the second sample set is a sample in the first sample set for the current round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0142] Alternatively, the computer-readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: perform a shuffling and replicated secret sharing operation of a first data set with a data holder and a model holder, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the participants of the pre-trained model fine-tuning include the data holder, the model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a first model after training on a corresponding sample, the first model after training is obtained by the data holder training the first model using at least part of the samples in the fine-tuning sample set; in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two first shards locally, the pre-trained model is fine-tuned with the data holder and the model holder in multiple rounds of iterative fine-tuning, so that each of the participants performs multiple rounds of iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines K samples with maximum reduced reserved loss RHO from a second sample set based on the two first shards of the second data set with the data holder and the model holder, and updates the model parameters using the K samples, the second sample set is a sample used in the current round of iterative fine-tuning in the first sample set, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0143] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0144] The diagrams of the flow and block in the drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flow or block diagram can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagram and / or flow diagram, and the combination of blocks in the block diagram and / or flow diagram, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0145] The modules described in the embodiments of the present disclosure can be implemented in the form of software or in the form of hardware. Among them, the name of the module does not constitute a limitation to the module itself in some cases, for example, the first acquisition module can also be described as "a module for acquiring the second shard that the model holder does not hold after fine-tuning from the data holder or the auxiliary computing party".

[0146] The functions described above in the present disclosure can be executed at least in part by one or more hardware logic components. For example, non-limiting example types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0147] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0148] According to one or more embodiments of the present disclosure, example 1 provides a method for fine-tuning a pre-trained model, participants for fine-tuning the pre-trained model include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the method is applied to the data holder, and the method comprises: training a first model using at least part of the fine-tuning sample set; for each sample in a first sample set, determining a first sample loss of the first model on the sample after training, wherein the first sample set is a subset of the fine-tuning sample set; performing a shuffling and replicated secret sharing operation of a first data set with the model holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set comprises the first sample set and the first sample loss of each sample in the first sample set; in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, performing multi-round iterative fine-tuning of the pre-trained model with the model holder and the auxiliary computing party based on the two first shards locally, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set locally with the model holder and the auxiliary computing party, and updates the model parameters using the K samples, the second sample set is the sample in the first sample set used for the current round of iterative fine-tuning, and the second data set comprises the second sample set and the first sample loss of each sample in the second sample set.

[0149] According to one or more embodiments of the present disclosure, example 2 provides the method of example 1, wherein the determining, by the model holder and the auxiliary computing party, the K samples with the maximum reducible reserved loss RHO from the second sample set based on the two first shards of the locally based second sample set, comprises: calculating, by the model holder and the auxiliary computing party, a second sample loss of each sample in the second sample set on the pre-trained model obtained after fine-tuning of the last round of iteration based on the two shards of the locally based second sample set, so that each of the participating parties obtains two third shards of the second sample loss respectively; calculating, by the model holder and the auxiliary computing party, the RHO of each sample in the second sample set according to the two shards of the locally based first sample loss of each sample in the second sample set and the two third shards, so that each of the participating parties obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; determining, by the model holder and the auxiliary computing party, the K samples with the maximum RHO from the second sample set based on the two fourth shards of the RHO of each sample in the second sample set locally.

[0150] According to one or more embodiments of the present disclosure, example 3 provides the method of example 2, wherein the determining, by the model holder and the auxiliary computing party, the K samples with the maximum RHO from the second sample set based on the two fourth shards of the RHO of each sample in the second sample set locally, comprises: sorting, by the model holder and the auxiliary computing party, the RHO of each sample in the second sample set based on the two fourth shards of the RHO of each sample in the second sample set locally, and determining the K samples with the maximum RHO from the second sample set according to the sorting result.

[0151] According to one or more embodiments of the present disclosure, example 4 provides the method of any one of examples 1-3, the performing, by the model holder and the helper computing party, a shuffling and replication secret sharing operation of the first data set so that each of the participating parties obtains two first shards of the first data set after shuffling, comprises: distributing the first data set to the model holder and the helper computing party in a two-party additive secret sharing manner, so that the model holder and the helper computing party obtain one fifth shard of the first data set respectively, and respectively shuffle the fifth shard held by oneself using a first random seed, generate two first shards of the first data set after shuffling based on the fifth shard after shuffling, and send the first shard of the two first shards which is not common to the model holder and the helper computing party to the data holder; receiving the first shard sent by the model holder and the helper computing party respectively as the two first shards of the first data set after shuffling.

[0152] According to one or more embodiments of the present disclosure, example 5 provides a fine-tuning method of a pre-trained model, participants of the fine-tuning of the pre-trained model include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the method is applied to the model holder, and the method comprises: performing a shuffling and replicated secret sharing operation of a first data set with the data holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, the first sample loss is a sample loss of a first model obtained after training on a corresponding sample, and the first model obtained after training is obtained by the data holder training a first model using at least part of the samples in the fine-tuning sample set; distributing model parameters of the pre-trained model to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants obtains two second shards of the model parameters; based on the two first shards locally, performing multi-round iterative fine-tuning of the pre-trained model with the data holder and the auxiliary computing party, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the model holder determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set with the data holder and the auxiliary computing party, and updates the model parameters using the K samples, the second sample set is a sample in the first sample set for the current round of iterative fine-tuning, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0153] According to one or more embodiments of the present disclosure, example 6 provides the method of example 5, the two first shards of the locally based second dataset, with the data holder and the auxiliary computing party determining K samples with maximum reduced retention loss RHO from the second sample set, comprising: based on the two shards of the locally based second sample set, with the data holder and the auxiliary computing party calculating the second sample loss of each sample in the second sample set on the pre-trained model obtained after fine-tuning of the last round of iteration, so that each of the participants obtains two third shards of the second sample loss respectively; according to the two shards of the first sample loss of each sample in the locally based second sample set and the two third shards, with the data holder and the auxiliary computing party calculating the RHO of each sample in the second sample set respectively, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; based on the two fourth shards of the RHO of each sample in the locally based second sample set, with the data holder and the auxiliary computing party determining K samples with maximum RHO from the second sample set.

[0154] According to one or more embodiments of the present disclosure, example 7 provides the method of example 6, the two fourth shards of the RHO of each sample in the locally based second sample set, with the data holder and the auxiliary computing party determining K samples with maximum RHO from the second sample set, comprising: based on the two fourth shards of the RHO of each sample in the locally based second sample set, with the data holder and the auxiliary computing party sorting the RHO of each sample in the second sample set by a multi-party secure sorting algorithm, and determining K samples with maximum RHO from the second sample set according to the sorting result.

[0155] According to one or more embodiments of the present disclosure, example 8 provides the method of example 5, wherein the performing, by the data holder and the auxiliary computing party, the shuffling and replication secret sharing operation of the first data set to enable each of the participating parties to obtain two first shards of the first data set after shuffling, comprises: in response to receiving one fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, shuffling the local fifth shard with a first random seed to obtain a sixth shard; generating a first random data set of the same specification as the sixth shard, and sending the first random data set to the auxiliary computing party; subtracting the sixth shard from the data at the corresponding positions of the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holder; and in response to receiving a second random data set sent by the auxiliary computing party, determining the sum of the first random data set and the second random data set as the first shard.

[0156] According to one or more embodiments of the present disclosure, example 9 provides the method of any one of examples 5-8, further comprising: obtaining, from the data holder or the auxiliary computing party, the second shard of the pre-trained model that is not held by the model holder after fine-tuning; and merging the second shard obtained from the data holder or the auxiliary computing party with the two second shards locally to obtain the model parameters of the pre-trained model after fine-tuning.

[0157] According to one or more embodiments of the present disclosure, example 10 provides a method for fine-tuning a pre-trained model, participants for fine-tuning the pre-trained model include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, the method is applied to the auxiliary computing party, and the method comprises: performing a shuffling and replicated secret sharing operation of a first data set with the data holder and the model holder, so that each of the participants obtains two first shards of the first data set after shuffling, wherein the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a first model obtained after training on a corresponding sample, the first model obtained after training is obtained by the data holder training a first model using at least part of the samples in the fine-tuning sample set; in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, based on the two first shards locally, performing multi-round iterative fine-tuning of the pre-trained model with the data holder and the model holder, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines K samples with maximum reducible reserved loss RHO from a second sample set based on the two first shards of the second data set locally together with the data holder and the model holder, and updates the model parameters by back propagation using the K samples, the second sample set is a sample for the current round of iterative fine-tuning in the first sample set, and the second data set includes the second sample set and the first sample loss of each sample in the second sample set.

[0158] According to one or more embodiments of the present disclosure, example 11 provides the method of example 10, wherein the performing, by the data holder and the model holder, a shuffling and replication secret sharing operation on the first data set to enable each of the participants to obtain two first shards of the first data set after shuffling, comprises: in response to receiving one fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, shuffling the local fifth shard with a first random seed to obtain a seventh shard; generating a second random data set of the same specification as the seventh shard, and sending the second random data set to the model holder; subtracting the seventh shard from the second random data set at corresponding positions to obtain a fourth data set, determining the fourth data set as the first shard, and sending the fourth data set as the first shard to the data holder; and in response to receiving the first random data set sent by the model holder, determining the sum of the first random data set and the second random data set as the first shard.

[0159] According to one or more embodiments of the present disclosure, example 12 provides an image classification method applied to a model holder, the method comprising: obtaining a first image to be classified; inputting the first image into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of the pre-trained model in any one of examples 1-11.

[0160] According to one or more embodiments of the present disclosure, example 13 provides a computer readable medium having stored thereon a computer program, which, when executed by a processing device, implements the steps of the method of any one of examples 1-12.

[0161] According to one or more embodiments of the present disclosure, example 14 provides an electronic device, comprising: a storage device having stored thereon a computer program; and a processing device configured to execute the computer program in the storage device to implement the steps of the method of any one of examples 1-12.

[0162] According to one or more embodiments of the present disclosure, example 15 provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any one of examples 1-12.

[0163] The above description merely illustrates the preferred embodiment of the disclosure and a principle of applied technologies. It should be understood by those skilled in the art that the disclosed range of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0164] Furthermore, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are contained in the above discussion, these should not be construed as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0165] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely illustrative of the example forms of implementing the claims. As to the means for performing the operations of the apparatus in the above-described embodiments, the specific manner in which the various modules perform the operations has been described in detail in the embodiments related to the method, and will not be described here in detail.

Claims

1. A fine-tuning method of a pre-trained model, characterized in that, The participants of the pre-trained model fine-tuning include a data holder holding a fine-tuning sample set, a model holder holding the pre-trained model, and an auxiliary computing party, the method is applied to the data holder, and the method comprises: training the first model by using at least part of the samples in the fine-tuning sample set; determining, for each sample in a first sample set, a first sample loss of the first model on the sample, wherein the first sample set is a subset of the fine-tuning sample set; performing, with the model holder and the auxiliary computing party, a shuffle and replicated secret sharing operation on a first data set, so that each of the participants obtains two first shards of the shuffled first data set, wherein the first data set comprises the first sample set and the first sample loss of each sample in the first sample set; in response to receiving two second shards of model parameters of the pre-trained model distributed by the model holder in a replicated secret sharing manner, performing, with the model holder and the auxiliary computing party, multi-round iterative fine-tuning on the pre-trained model based on the two local first shards, so that each of the participants performs multi-round iterative fine-tuning on the two local second shards until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the data holder determines, with the model holder and the auxiliary computing party, K samples with the maximum reducible reserved loss RHO from a second sample set based on two first shards of a second data set, and updates the model parameters by using the K samples for back propagation, the second sample set being the samples in the first sample set used for the current round of iterative fine-tuning, and the second data set comprising the second sample set and the first sample loss of each sample in the second sample set.

2. The method of claim 1, wherein, The determination of the K samples with the maximum reducible reserved loss RHO from the second sample set based on the two local first shards of the second data set comprises: calculating, with the model holder and the auxiliary computing party, the second sample loss of each sample in the second sample set on the pre-trained model obtained after the last round of iterative fine-tuning based on two shards of the local second sample set, so that each of the participants obtains two third shards of the second sample loss; calculating, with the model holder and the auxiliary computing party, the RHO of each sample in the second sample set according to the two shards of the first sample loss of each sample in the local second sample set and the two local third shards, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; determining, with the model holder and the auxiliary computing party, the K samples with the maximum RHO from the second sample set based on the two local fourth shards of the RHO of each sample in the second sample set.

3. The method of claim 2, wherein, The two fourth shares of the RHO of each sample in the second sample set based on the local, and the model holder and the auxiliary computing party determine the K samples with the maximum RHO from the second sample set, including: The two fourth shares of the RHO of each sample in the second sample set based on the local, and the model holder and the auxiliary computing party sort the RHO of each sample in the second sample set through a multi-party secure sorting algorithm, and determine the K samples with the maximum RHO from the second sample set according to the sorting result.

4. The method according to any one of claims 1 to 3, characterized in that, The model holder and the auxiliary computing party perform a shuffle and replicated secret sharing operation on the first data set, so that each of the participants obtains two first shares of the first data set after shuffling, including: The first data set is distributed to the model holder and the auxiliary computing party in a two-party additive secret sharing manner, so that the model holder and the auxiliary computing party obtain a fifth share of the first data set, respectively, and use a first random seed to shuffle the fifth share held by themselves, generate two first shares of the first data set after shuffling based on the fifth share after shuffling, and send the first share that is not common to the model holder and the auxiliary computing party to the data holder; Receive the first share sent by the model holder and the auxiliary computing party, respectively, as two first shares of the first data set after shuffling.

5. A fine-tuning method of a pre-trained model, characterized in that, The participants of the pre-trained model fine-tuning include a data holder, a model holder and an auxiliary computing party, the data holder holds a fine-tuning sample set, the model holder holds the pre-trained model, and the method is applied to the model holder, and the method includes: The model holder and the auxiliary computing party perform a shuffle and replicated secret sharing operation on the first data set, so that each of the participants obtains two first shares of the first data set after shuffling, including: the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a first model after training on a corresponding sample, and the first model after training is obtained by training a first model by the data holder using at least part of the samples in the fine-tuning sample set; The model parameters of the pre-trained model are distributed to the data holder and the auxiliary computing party in a replicated secret sharing manner, so that each of the participants obtains two second shares of the model parameters; The model holder and the auxiliary computing party perform a shuffle and replicated secret sharing operation on the first data set, so that each of the participants obtains two first shares of the first data set after shuffling, including: the first data set includes a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a first model after training on a corresponding sample, and the first model after training is obtained by training a first model by the data holder using at least part of the samples in the fine-tuning sample set; performing multi-round iterative fine-tuning on the pre-trained model with the data holder and the auxiliary computing party based on the two first shards of the local second dataset, wherein in each round of iterative fine-tuning, the model holder determines, from a second sample set, K samples with maximum reducible reserved loss RHO based on the two first shards of the local second dataset and the data holder and the auxiliary computing party, and updates the model parameters by back propagation using the K samples, the second sample set being the samples in the first sample set used for the current round of iterative fine-tuning, and the second dataset including the second sample set and the first sample loss of each sample in the second sample set.

6. The method of claim 5, wherein, The determining, from the second sample set, K samples with maximum reducible reserved loss RHO based on the two first shards of the local second dataset and the data holder and the auxiliary computing party, comprises: performing, based on the two shards of the local second sample set, calculation of the second sample loss of each sample in the second sample set on the pre-trained model obtained after the last round of iterative fine-tuning with the data holder and the auxiliary computing party, so that each of the participants obtains two third shards of the second sample loss respectively; calculating, according to the two shards of the first sample loss of each sample in the local second sample set and the two third shards, the RHO of each sample in the second sample set with the data holder and the auxiliary computing party, so that each of the participants obtains two fourth shards of the RHO of each sample in the second sample set respectively, wherein the RHO of the sample is equal to the difference between the second sample loss of the sample and the first sample loss of the sample; determining, based on the two fourth shards of the RHO of each sample in the local second sample set, K samples with maximum RHO from the second sample set with the data holder and the auxiliary computing party.

7. The method of claim 6, wherein, The determining, based on the two fourth shards of the RHO of each sample in the local second sample set, K samples with maximum RHO from the second sample set with the data holder and the auxiliary computing party, comprises: performing, based on the two fourth shards of the RHO of each sample in the local second sample set, sorting of the RHO of each sample in the second sample set by a multi-party secure sorting algorithm with the data holder and the auxiliary computing party, and determining, according to the sorting result, K samples with maximum RHO from the second sample set.

8. The method of claim 5, wherein, The performing, with the data holder and the auxiliary computing party, shuffling and replicated secret sharing operation on the first dataset, so that each of the participants obtains two first shards of the first dataset obtained after shuffling, comprises: performing shuffling on the local fifth shard using a first random seed in response to receiving one fifth shard of the first dataset distributed by the data holder in a two-party additive secret sharing manner, to obtain a sixth shard; performing, based on the two first shards of the local first dataset, shuffling and replicated secret sharing operation on the first dataset with the data holder and the auxiliary computing party, so that each of the participants obtains two first shards of the first dataset obtained after shuffling, comprises: performing shuffling on the local fifth shard using a first random seed in response to receiving one fifth shard of the first dataset distributed by the data holder in a two-party additive secret sharing manner, to obtain a sixth shard; generating a first random data set same as the sixth shard specification, and sending the first random data set to the auxiliary computing party; subtracting data at corresponding positions of the sixth shard and the first random data set to obtain a third data set, determining the third data set as the first shard, and sending the third data set as the first shard to the data holding party; in response to receiving a second random data set sent by the auxiliary computing party, determining a sum of the first random data set and the second random data set as the first shard.

9. The method according to any one of claims 5-8, characterized in that, The method further comprises: obtaining the second shard obtained after fine-tuning from the data holding party or the auxiliary computing party, which is not held by the model holding party; merging the second shard obtained from the data holding party or the auxiliary computing party with the two second shards locally to obtain the model parameters of the pre-trained model obtained after fine-tuning.

10. A fine-tuning method of a pre-trained model, characterized in that, The participants in the fine-tuning of the pre-trained model include a data holding party, a model holding party and an auxiliary computing party, the data holding party holds a fine-tuning sample set, the model holding party holds the pre-trained model, and the method is applied to the auxiliary computing party, and the method comprises: performing a shuffle and replicated secret sharing operation of a first data set with the data holding party and the model holding party, so that each of the participants obtains two first shards of the first data set obtained after shuffling, wherein the first data set comprises a first sample set and a first sample loss of each sample in the first sample set, the first sample set is a subset of the fine-tuning sample set, and the first sample loss is a sample loss of a first model obtained after training on a corresponding sample, wherein the first model obtained after training is obtained by the data holding party training a first model using at least part of the samples in the fine-tuning sample set; in response to receiving two second shards of the model parameters of the pre-trained model distributed in a replicated secret sharing manner by the model holding party, performing multi-round iterative fine-tuning of the pre-trained model with the data holding party and the model holding party based on the two first shards locally, so that each of the participants performs multi-round iterative fine-tuning on the two second shards locally until a fine-tuning cutoff condition is reached, wherein in each round of iterative fine-tuning, the auxiliary computing party determines K samples with maximum reserved loss RHO from a second sample set based on the two first shards of the second data set with the data holding party and the model holding party, and updates the model parameters by back propagation using the K samples, wherein the second sample set is a sample used in the current round of iterative fine-tuning in the first sample set, and the second data set comprises the second sample set and the first sample loss of each sample in the second sample set.

11. The method of claim 10, wherein, The shuffle and replicated secret sharing operation of the first data set with the data holding party and the model holding party to enable each of the participants to obtain two first shards of the first data set obtained after shuffling, comprises: In response to receiving the fifth shard of the first data set distributed by the data holder in a two-party additive secret sharing manner, the local fifth shard is shuffled using a first random seed to obtain a seventh shard; A second random data set of the same specification as the seventh shard is generated, and the second random data set is sent to the model holder; The seventh shard is subtracted from the data at the corresponding position of the second random data set to obtain a fourth data set, the fourth data set is determined as the first shard, and the fourth data set is sent to the data holder as the first shard; In response to receiving the first random data set sent by the model holder, the sum of the first random data set and the second random data set is determined as the first shard.

12. An image classification method, characterized by, Applied to the model holder, the method comprises: Obtaining a first image to be classified; Inputting the first image into a target image classification model to obtain a classification result of the first image, wherein the target image classification model is obtained by fine-tuning an initial image classification model according to the fine-tuning method of the pre-training model in any one of claims 1-11, and the initial image classification model is a pre-training model.

13. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by the processing device to realize the steps of the method of any one of claims 1-12.

14. An electronic device, comprising: Comprise: A storage device having a computer program stored thereon; A processing device for executing the computer program in the storage device to realize the steps of the method of any one of claims 1-12.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1-12. The computer program is executed by the processor to realize the steps of the method of any one of claims 1-12.

Images