Model training method and device based on secure multi-party computation, equipment and medium

Abstract

The application discloses a model training method and device based on secure multi-party computation, equipment and medium. The model training method based on secure multi-party computation comprises the following steps: acquiring an initial model corresponding to a current node, the initial models corresponding to all nodes in a secure multi-party computation system being the same; inputting sample data matched with the current node into the initial model corresponding to the current node for model training, so as to obtain a trained model; determining model update parameters corresponding to the current node based on model parameters in the trained model corresponding to the current node and model parameters in trained models corresponding to other nodes, the secure multi-party computation system comprising the current node and the other nodes; and updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node. The above scheme can improve the training efficiency of the model training process.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a model training method, apparatus, device, and medium based on secure multi-party computation. Background Technology

[0002] In recent years, deep learning has been widely used in various fields, but deep learning requires a large amount of data to train models.

[0003] However, this data is usually owned by different entities, and because this data may involve privacy and trade secrets, the entities are usually unwilling to share it due to data security concerns, which will greatly reduce the training effect of the neural network model.

[0004] Therefore, improving the training efficiency of the model training process while ensuring data security has become an important challenge. Summary of the Invention

[0005] This application provides at least one model training method, apparatus, device, and computer-readable storage medium based on secure multi-party computation.

[0006] This application provides a model training method based on secure multi-party computation, comprising: obtaining an initial model corresponding to a current node, wherein the initial models corresponding to each node in the secure multi-party computation system are identical; inputting the obtained sample data matching the current node into the initial model corresponding to the current node for model training to obtain a trained model; determining model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes, wherein the secure multi-party computation system includes the current node and the other nodes; and updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node.

[0007] In one embodiment, the step of determining the model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes, wherein the secure multi-party computation system includes the current node and other nodes, includes: aggregating the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes to obtain aggregated parameters; and decrypting the model update parameters corresponding to the current node from the aggregated parameters according to the key corresponding to the current node.

[0008] In one embodiment, the step of aggregating the model parameters in the trained model corresponding to the current node with the model parameters in the trained models corresponding to other nodes to obtain aggregated parameters includes: encrypting the model parameters in the trained model based on the key corresponding to the current node to obtain encrypted model parameters; and merging the encrypted model parameters corresponding to each node to obtain aggregated parameters.

[0009] In one embodiment, the step of inputting the acquired sample data matching the current node into the initial model corresponding to the current node for model training to obtain the trained model includes: obtaining gradient update parameters during model training based on the sample data matching the current node; calculating the average value of the gradient update parameters among the nodes and using the average value as the average gradient update parameter; updating the model parameters of the initial model corresponding to the current node with the average gradient update parameter to obtain the trained model and the model parameters of the trained model.

[0010] In one embodiment, before inputting the acquired sample data matching the current node into the initial model corresponding to the current node for model training to obtain the trained model, the method further includes: selecting a subset of sample data corresponding to the current node from a preset sample dataset; if there is identical sample data between the subset of sample data corresponding to the current node and the subset of sample data corresponding to other nodes, then performing deduplication processing on the data in the subset of sample data corresponding to the current node or the subset of sample data corresponding to other nodes; and using the data in the deduplicated subset of sample data as the sample data matching the current node or the sample data matching other nodes.

[0011] In one embodiment, the step of obtaining the initial model corresponding to the current node includes: building a model based on preset model building data to obtain a basic model; initializing the basic model according to the model building data to obtain an initialized basic model; and synchronizing the initialized basic model corresponding to the current node with the initialized basic models corresponding to other nodes to obtain the initial model corresponding to the current node.

[0012] In one embodiment, after the step of updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node, the method further includes: evaluating the updated model corresponding to the current node; if the updated model meets the preset evaluation conditions, pausing model training; if the updated model does not meet the preset evaluation conditions, continuing model training.

[0013] A second aspect of this application provides a model training apparatus based on secure multi-party computation, comprising: an acquisition module for acquiring an initial model corresponding to a current node, wherein the initial models corresponding to each node in the secure multi-party computation system are identical; a model training module for inputting the acquired sample data matching the current node into the initial model corresponding to the current node for model training to obtain a trained model; a calculation module for determining model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes, wherein the secure multi-party computation system includes the current node and the other nodes; and an update module for updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node.

[0014] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described model training method based on secure multi-party computation.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described model training method based on secure multi-party computation.

[0016] The above scheme inputs the sample data matching the current node into the initial model corresponding to the current node for model training, resulting in the trained model corresponding to the current node. Since the initial models corresponding to each node are the same, the model update parameters for updating the initial model corresponding to the current node can be determined based on the model parameters of the trained model corresponding to the current node and the trained model parameters corresponding to other nodes. The initial model corresponding to the current node is then updated based on the model update parameters. This avoids disclosing the sample data and neural network models of each node, thereby improving the training efficiency of the model training process while ensuring data security.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a flowchart illustrating an exemplary embodiment of the model training method based on secure multi-party computation of this application; Figure 2This is a flowchart illustrating how the model update parameters corresponding to the current node are determined in the model training method based on secure multi-party computation of this application. Figure 3 This is a flowchart illustrating how the model training method based on secure multi-party computation of this application inputs the obtained sample data matching the current node into the initial model corresponding to the current node for model training, and obtains the trained model. Figure 4 This is a block diagram illustrating a model training apparatus based on secure multi-party computation, as shown in an exemplary embodiment of this application. Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the model training method based on secure multi-party computation according to this application. This method can be applied to nodes in a secure multi-party computation system. Specifically, it may include the following steps: Step S110: Obtain the initial model corresponding to the current node. The initial models corresponding to each node in the secure multi-party computation system are the same.

[0024] A secure multi-party computation system comprises two or more nodes. All or some of these nodes act as participants in secure multi-party computation and model training. For ease of understanding and explanation, the nodes mentioned below are typically those acting as participants, and may also be referred to as participants or participating nodes. It should be noted that the secure multi-party computation-based model training method provided in this application can be applied synchronously or asynchronously to any participant in the secure multi-party computation system.

[0025] The initial model can refer to either a newly initialized model or a pre-trained model that has undergone a certain degree of training. There is no limitation here. Each participant has its own corresponding initial model, and the initial models of each participant are the same to ensure that the data of each participant can be referenced during the model training process, so that the trained model is more accurate and the model training process can be accelerated.

[0026] Step S120: Input the obtained sample data matching the current node into the initial model corresponding to the current node for model training to obtain the trained model.

[0027] Sample data refers to the data used for model training. Each participant has corresponding matching sample data. The sample data of different participants may have the same parts or may be completely different. However, the sample data used by each participant when training the model is matched with its own node, so as to ensure that each node uses its own sample data for model training without having to share its data with other participants.

[0028] The trained model can be a converged model that has completed training, or a non-converged model that has been iterated a certain number of times; there is no limitation here.

[0029] Specifically, each participant will obtain sample data that matches its own. That is, for the current participant's node, if the current node obtains sample data that matches its own, the current node will input the obtained sample data that matches its own into the initial model corresponding to the current node for model training, and obtain the trained model.

[0030] Step S130: Based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes, determine the model update parameters corresponding to the current node. The secure multi-party computation system includes the current node and other nodes.

[0031] Model update parameters are used to update the neural network model. It should be noted that the model update parameters for each node may be the same or different.

[0032] In a secure multi-party computation system, the current node is the current participant, and other nodes are the other participants.

[0033] Specifically, in order to further optimize the trained model and improve its accuracy, the same or different sample data are used to train the model on the same initial model in the nodes of each participant, and the model update parameters for updating the trained model for each node are determined according to the model parameters of the trained model corresponding to each node.

[0034] Step S140: Update the initial model corresponding to the current node according to the model update parameters corresponding to the current node.

[0035] For example, the model parameters of the initial model corresponding to the current node are updated using the model update parameters corresponding to the current node to improve the model accuracy. This achieves the goal of improving model training efficiency and eliminating the need to disclose the sample data used for training when each node trains the same initial model through multi-party computation.

[0036] As can be seen, this application inputs the sample data matching the current node into the initial model corresponding to the current node for model training, and obtains the trained model corresponding to the current node. Since the initial models corresponding to each node are the same, the model update parameters for updating the initial model corresponding to the current node can be determined based on the model parameters of the trained model corresponding to the current node and the trained model parameters of other nodes. The initial model corresponding to the current node is updated based on the model update parameters of the current node. This avoids disclosing the sample data and neural network model of each node, and improves the training efficiency of the model training process while ensuring data security.

[0037] Based on the above embodiments, the embodiments of this application adopt... Figure 2 The flowchart shown details how to determine the model update parameters for the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes. Specifically, the method in this embodiment includes the following steps: Step S210: Aggregate the model parameters in the trained model corresponding to the current node with the model parameters in the trained models corresponding to other nodes to obtain aggregated parameters.

[0038] Aggregation processing refers to the aggregation of model parameters from the trained model of the current node and the trained models of other nodes based on a preset computation protocol. In other words, it involves performing aggregation calculations based on the model parameters of the trained models of each node to obtain aggregated parameters. Aggregation processing can also include encrypted aggregation processing, where each node encrypts its own trained model parameters before performing aggregation calculations, obtaining encrypted model parameters for each node, and then performing aggregation calculations based on the encrypted model parameters of each node to obtain aggregated parameters.

[0039] For example, the preset computation protocol can be a Secure Multi-Party Computation (MPC) protocol. Secure Multi-Party Computation is a type of privacy-preserving computation technology that allows a group of participants to jointly compute a function (equivalent to the model training process in this application) using their private data as input. All participants can only obtain the output of this function, i.e., the aggregated parameters, and cannot obtain any other information.

[0040] Furthermore, regarding the secure multi-party computation protocol, methods for encrypting data include, but are not limited to, using a secure multi-party computation algorithm based on secret sharing technology. Specifically, this algorithm uses secret sharing technology to encrypt the data separately and distribute it to each participating party, obtaining the final result through multi-party computation.

[0041] Alternatively, the method for encrypting data can also be a secure multi-party computation algorithm based on homomorphic encryption. Specifically, this algorithm uses homomorphic encryption technology to encrypt the data, performs computation in the ciphertext space, and finally outputs the ciphertext of the result, which participants can then decrypt to obtain the result.

[0042] Alternatively, the method for encrypting the data can also be a secure multi-party computation algorithm based on differential privacy. Specifically, this algorithm uses differential privacy technology to encrypt the data and adds noise during the computation to protect privacy.

[0043] Step S220: Decrypt the model update parameters corresponding to the current node from the aggregate parameters according to the key corresponding to the current node.

[0044] The key is randomly generated by each node. The key can be a public-private key pair used to encrypt and decrypt data content in the secure multi-party computation process. For example, the key corresponding to each node can be used to encrypt and decrypt the model parameters of the initial model and / or the model parameters of the trained model corresponding to each node. The key corresponding to each node can also be used to encrypt the matching sample data obtained by each node. It should be noted that this application does not limit the number of keys generated by each node. Each node holding its own corresponding key avoids data leakage and improves the security and privacy of the model training process.

[0045] Optionally, the key for each node can be generated before each node obtains the corresponding initial model.

[0046] As can be seen, this application encrypts relevant data such as sample data and / or model parameters of each node during the model training process through a secure multi-party computation protocol and data encryption. It then performs calculations and aggregations based on the encrypted data to obtain aggregated parameters, thereby avoiding the security problem of data leakage caused by data sharing during the model training process, ensuring the security of the model training process, and thus improving the training effect of the model.

[0047] Based on the above embodiments, this embodiment explains the steps of aggregating the model parameters in the trained model corresponding to the current node with the model parameters in the trained models corresponding to other nodes to obtain aggregated parameters. Specifically, it includes: The model parameters in the trained model are encrypted using the key corresponding to the current node to obtain the encrypted model parameters; the encrypted model parameters corresponding to each node are then merged to obtain the aggregate parameters.

[0048] Aggregate parameters refer to the model parameters of a new global model calculated jointly based on the encrypted model parameters corresponding to each node under a secure multi-party computation protocol. It can be understood that the aggregate parameters obtained from the encrypted model parameters are also encrypted.

[0049] Specifically, the encryption process can refer to one or more encryption calculation methods provided in the above embodiments. The encryption process and the aggregation process are calculated based on a secure multi-party computation protocol. The encrypted model parameters corresponding to each node are aggregated to obtain the aggregate parameters.

[0050] Optionally, after obtaining the encrypted model parameters, the encrypted model parameters corresponding to each node can be sent to the aggregation node. The aggregation node then aggregates the encrypted parameters from each node to obtain the model parameters of a global model, and returns these global model parameters to each node. This allows each node to update its initial model based on the global model parameters. The aggregation node can be a participating node in the secure multi-party computation system, or it can be a node that is not a participating node. This is not limited to any particular node and can be determined based on the actual computing power of each node.

[0051] Therefore, by encrypting the model parameters in the trained model corresponding to each node, the encrypted model parameters corresponding to each node are obtained. Then, the encrypted model parameters corresponding to each node are aggregated to obtain the aggregated parameters jointly trained by each node. This avoids data leakage during the model training process and improves the security of model training.

[0052] Based on the above embodiments, the embodiments of this application adopt... Figure 3 The flowchart shown details the steps of inputting the acquired sample data matching the current node into the initial model corresponding to the current node for model training, thereby obtaining the trained model. Specifically, the method in this embodiment includes the following steps: Step S310: Obtain gradient update parameters during model training based on the sample data matched by the current node.

[0053] Gradient update, including gradient descent, can be considered an optimization algorithm that updates model parameters to find the minimum of the loss function. By continuously training the model and obtaining gradient update parameters, the model parameters are updated, which makes the output of the neural network model more accurate. This is also the goal and significance of model training.

[0054] Specifically, after the sample data matching the current node is obtained and input into the initial model corresponding to the current node, the initial model corresponding to the current node will be trained based on the sample data, and one or more gradient update parameters will be obtained during the model training process.

[0055] For example, backpropagation, or BP algorithm, can be used during model training. By using the residual between the predicted value and the label value, the gradient of the loss function with respect to the network parameters of each layer can be solved backward from the output layer to the input layer, and the gradient can be used to update the parameters.

[0056] Step S320: Calculate the average value of the gradient update parameters of each node, and use the average value of the parameters as the average gradient update parameter.

[0057] Specifically, during the model training process at each node, the average gradient update parameter is obtained by calculating the average value based on the secure multi-party computation protocol and the gradient update parameters corresponding to each node.

[0058] Step S330: Update the model parameters of the initial model corresponding to the current node using the average gradient update parameters to obtain the trained model and the model parameters of the trained model.

[0059] Specifically, each node updates the model parameters of its corresponding initial model according to the average gradient update parameters, and uses a secure multi-party computation protocol to jointly perform the computation to ensure that the model parameters of each participating node are the same. This includes the current node updating the model parameters of its corresponding initial model with the average gradient update parameters to obtain the trained model and the model parameters of the trained model.

[0060] Based on the above embodiments, this application embodiment describes the process prior to the step of inputting the obtained sample data matching the current node into the initial model corresponding to the current node for model training to obtain the trained model. Specifically, the method of this embodiment includes the following steps: Select a subset of sample data corresponding to the current node from the preset sample dataset; if there is duplicate sample data between the subset of sample data corresponding to the current node and the subset of sample data corresponding to other nodes, then perform deduplication on the data in the subset of sample data corresponding to the current node or the subset of sample data corresponding to other nodes; use the data in the deduplicated subset of sample data as the sample data matching the current node or the sample data matching other nodes.

[0061] A sample dataset refers to a collection of sample data used for model training, which includes one or more subsets of sample data.

[0062] A sample data subset refers to a subset of data selected from a sample dataset. Different participants may select the same data from the sample dataset, so that different sample data subsets may contain the same sample data. This is equivalent to the sample data subsets corresponding to different participants having a mathematical intersection.

[0063] It's important to note that if different participants use overlapping data for model training—that is, if the sample data subset corresponding to the current node contains the same data as the sample data subsets corresponding to other nodes—this could lead to data leakage during model training based on secure multi-party computation. For example, if participant A and participant B have the same sample data, participant A's model parameters and sample data information could be compromised by participant B, and vice versa. Therefore, it is necessary to deduplicate the duplicate sample data between different nodes.

[0064] To further explain, deduplication can be performed by removing duplicate data from the subset of sample data corresponding to the current node or by removing duplicate data from the subset of sample data corresponding to other nodes. The goal is simply to ensure that the data in the subsets of sample data corresponding to each node are not duplicated.

[0065] It should also be noted that, in the process of selecting a subset of sample data corresponding to the current node from the preset sample dataset, the size and distribution of the sample data subsets obtained by different participants should be kept as balanced as possible to avoid uneven data processing and model bias. During the data selection process, some data encryption and anonymization measures can also be taken to ensure data security. For example, homomorphic encryption and differential privacy techniques can be used to encrypt data, and anonymization techniques can be used to hide sensitive information. In addition, in real-world scenarios, since the sample dataset is usually dynamically updated, it is necessary to update the sample data subset periodically.

[0066] Optionally, during the process of each node selecting a subset of sample data corresponding to its current node from a preset sample dataset, this can be viewed as each node performing data segmentation on the sample dataset to obtain the required subset of sample data. The data segmentation process includes: random segmentation, randomly dividing the sample dataset into multiple parts and then assigning each part to different participants, which ensures that the distribution and characteristics of the dataset obtained by each participant are relatively similar, thereby reducing model bias; and feature-based segmentation, dividing the sample data in the sample dataset into multiple parts according to the characteristics of the sample data, such as segmentation based on data source, data type, data label, etc., and then assigning each part to different participants. Each participant selects a corresponding subset of the sample data to ensure that the subset obtained by each participant has a certain degree of representativeness and characteristics, thereby improving the generalization ability of the model. The data is segmented based on the characteristics of the participants, and the data is divided according to factors such as the participants' capabilities, security requirements, and data sources. For example, participants with strong computing resources select a large-scale subset of the sample data from the sample dataset, while participants with security capabilities select a subset of the sample data containing sensitive data. This maximizes the use of the participants' advantages and characteristics, and improves the efficiency and security of data processing.

[0067] Based on the above embodiments, this application embodiment describes the steps for obtaining the initial model corresponding to the current node. Specifically, the method of this embodiment includes the following steps: The model is built based on the preset model building data to obtain the basic model; the basic model is initialized according to the model building data to obtain the initialized basic model; the initialized basic model corresponding to the current node is synchronized with the initialized basic models corresponding to other nodes to obtain the initial model corresponding to the current node.

[0068] Model building data includes neural network structure, number and width of neural network layers, activation function, structural loss function, and optimizer.

[0069] Model initialization includes, but is not limited to, random initialization, fixed initialization, pre-training initialization, and custom initialization. It's important to note that neural networks optimize highly complex nonlinear models with virtually no global optimum, making model initialization crucial. The choice of initial point can sometimes determine whether the algorithm converges. When convergence occurs, the initial point can influence how quickly the learning converges and whether it reaches a point with high or low cost. Excessive initialization leads to gradient explosion, while insufficient initialization results in gradient vanishing. Therefore, selecting an initialization method that matches the model's data (such as the neural network structure and activation function) is essential to avoid gradient vanishing or exploding, thereby improving model training performance.

[0070] Optionally, after obtaining the initialized basic model, techniques such as model distillation can be used to compress the initialized basic model, reduce the model size and computational load, thereby improving computational performance.

[0071] Furthermore, after obtaining the initialized base model or the compressed base model, it is necessary to synchronize the models of each node. This is equivalent to synchronizing the initialized base model corresponding to the current node with the initialized base models corresponding to other nodes, as described above, to obtain the initial model corresponding to the current node. This ensures that the model parameters of the initial model held by each node are the same, thereby enabling each node to train the initial model together. Compared to the situation where each node holds different initial model parameters, this avoids the need to merge the model parameters of each node to generate the final model parameters, reduces additional communication and computation, and improves model training efficiency.

[0072] It should also be noted that model synchronization methods include, but are not limited to, full synchronization. In full synchronization, all participants need to wait for other participants to complete their calculations before proceeding to the next step. This ensures the correctness of the calculations but increases synchronization overhead and reduces computational efficiency. Asynchronous synchronization allows each participant to perform calculations independently, synchronizing only when results are available. This reduces synchronization overhead and improves computational efficiency, but may lead to synchronization inconsistencies, requiring a coordination mechanism to resolve. Parameter server synchronization stores the initial model's parameters on a parameter server, and each node accesses the parameter server via the network to update and retrieve the initial model's parameters. The advantage of the parameter server approach is its lower synchronization overhead, which can accelerate the training process.

[0073] Optionally, for secure multi-party computation, since data transmission involves multiple participants, compression algorithms such as LZ77, LZW, and Huffman can be used to reduce data transmission volume and improve computational efficiency. Compression methods include matrix compression. Matrices are a common data structure in secure multi-party computation; matrix compression techniques can compress matrix data into smaller data packets and decompress them during transmission. Matrix compression techniques include row compression, column compression, and diagonal compression. Compression formats, such as JPEG and MPEG, are common multimedia data formats for images and videos, and they employ specific compression formats to compress raw data into smaller data packets and decompress them during transmission. Data sampling can reduce data volume in some cases; for example, in image processing, downsampling can reduce the number of pixels, thus reducing data transmission volume. Data aggregation can also reduce data volume in secure multi-party computation by merging multiple data packets into a single data packet for transmission. For example, data from multiple participants can be aggregated into a single data packet for transmission.

[0074] Based on the above embodiments, this application embodiment describes the process after the step of updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node. Specifically, the method of this embodiment includes the following steps: Evaluate the updated model corresponding to the current node; if the updated model meets the preset evaluation conditions, pause model training; if the updated model does not meet the preset evaluation conditions, continue model training.

[0075] Model evaluation methods include, but are not limited to, using evaluation metrics. Choosing appropriate evaluation metrics to assess the model—for example, in classification tasks, metrics such as accuracy, precision, recall, and F1 score can be used; in regression tasks, metrics such as mean squared error (MSE) and mean absolute error (MAE) can be used; cross-validation is used to evaluate model performance. By dividing the dataset into several folds, each fold can be used to train the model, and the remaining folds can be used for validation. This helps reduce overfitting and improve the model's generalization ability; error analysis helps identify problems in the model and provides a basis for further optimization. The process of model training provides guidance, and various visualization tools can be used to analyze model errors, such as scatter plots, residual plots, and confusion matrices. Ensemble methods can be used to improve model performance and accuracy. For example, techniques such as bagging, boosting, and random forests can be used to integrate multiple models, thereby reducing the risk of overfitting and improving model accuracy. Hyperparameters, which are parameters set before training the model, such as learning rate and regularization parameters, can be optimized to improve model performance. Techniques such as grid search, random search, and Bayesian optimization can be used to find the optimal combination of hyperparameters.

[0076] Optionally, the model training method based on secure multi-party computation provided in this application embodiment may be to evaluate the updated model corresponding to the current node, or to evaluate the model after aggregating the participating party nodes in the secure multi-party computation system. Similarly, the above-mentioned corresponding operations are performed according to the evaluation results and preset evaluation conditions.

[0077] Based on the above embodiments, this application provides a supplementary description of the model training method based on secure multi-party computation. Specifically, the method of this embodiment may further include the following steps: Based on the Secure Boosting method, weak classifiers can be set in each participant. The sample data selected by each participant, i.e., the subset of sample data, is encrypted. Then, the encrypted data corresponding to each participant is input into the weak classifier corresponding to that participant to train the weak classifier. The results of the output of the weak classifiers corresponding to each participant are summarized and weighted to generate the final prediction result.

[0078] Specifically, the basic model is initialized, and an initial model is set. Weight distribution Number of iterations For each participating party, based on the weight distribution Sampling with replacement is performed from the sample dataset, and each participant will receive a subset of sample data corresponding to their own participant, which is used to train the initial model for each participant. ; Calculate the initial model for each participant. Error rate on a subset of sample data and the initial model weights for each participant. The weight distribution of the initial model for each participant is updated based on the error rate and weight coefficients of the initial model for each participant. Each participant i will use the trained initial model The encrypted model is sent to the aggregation node, which then aggregates the encrypted models to obtain a new global model. The aggregation node will display the global model. The new global model is sent to each participant, who then decrypts and evaluates it, calculating the error rate. Determine if the error rate meets the preset evaluation criteria; if so, pause model training and return to the global model. Otherwise, continue model training and proceed to the next iteration. It should be noted that the number of iterations can also be considered as an evaluation criterion; that is, if the number of iterations is greater than or equal to the preset number of iterations, model training should also be paused, and the global model should be returned. .

[0079] It should be further noted that the execution entity of the secure multi-party computation-based model training method can be a secure multi-party computation-based model training device. For example, the secure multi-party computation-based model training method can be executed by a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this secure multi-party computation-based model training method can be implemented by the processor calling computer-readable instructions stored in memory.

[0080] Figure 4 This is a block diagram illustrating a model training apparatus based on secure multi-party computation, as shown in an exemplary embodiment of this application. Figure 4 As shown, the exemplary model training apparatus 400 based on secure multi-party computation includes: an acquisition module 410, a model training module 420, a computation module 430, and an update module 440. Specifically: The acquisition module 410 is used to acquire the initial model corresponding to the current node. The initial models corresponding to all nodes in the secure multi-party computation system are the same. The model training module 420 is used to input the sample data that matches the current node into the initial model corresponding to the current node for model training, so as to obtain the trained model. The calculation module 430 is used to determine the model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes. The secure multi-party computation system includes the current node and other nodes. The update module 440 is used to update the initial model corresponding to the current node based on the model update parameters corresponding to the current node.

[0081] In this exemplary model training device based on secure multi-party computation, sample data matching the current node is input into the initial model corresponding to the current node for model training, resulting in the trained model corresponding to the current node. Since the initial models corresponding to each node are the same, the model update parameters for updating the initial model corresponding to the current node can be determined based on the model parameters of the trained model corresponding to the current node and the trained model parameters of other nodes. The initial model corresponding to the current node is updated based on the model update parameters, thereby avoiding the public disclosure of the sample data and neural network models of each node, and improving the training efficiency of the model training process while ensuring data security.

[0082] The functions of each module can be found in the implementation examples of the model training method based on secure multi-party computation, and will not be repeated here.

[0083] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 500 includes a memory 510 and a processor 520. The processor 520 is used to execute program instructions stored in the memory 510 to implement the steps in any of the above embodiments of the model training method based on secure multi-party computation. In a specific implementation scenario, the electronic device 500 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 500 may also include mobile devices such as laptops and tablets, which are not limited here.

[0084] Specifically, processor 520 controls itself and memory 510 to implement the steps in any of the above embodiments of the model training method based on secure multi-party computation. Processor 520 may also be referred to as a CPU (Central Processing Unit). Processor 520 may be an integrated circuit chip with signal processing capabilities. Processor 520 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 520 may be implemented using integrated circuit chips.

[0085] The above scheme inputs the sample data matching the current node into the initial model corresponding to the current node for model training, resulting in the trained model corresponding to the current node. Since the initial models corresponding to each node are the same, the model update parameters for updating the initial model corresponding to the current node can be determined based on the model parameters of the trained model corresponding to the current node and the trained model parameters corresponding to other nodes. The initial model corresponding to the current node is then updated based on the model update parameters. This avoids disclosing the sample data and neural network models of each node, thereby improving the training efficiency of the model training process while ensuring data security.

[0086] Please see Figure 6 , Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 610 stores program instructions 611 that can be executed by a processor. The program instructions 611 are used to implement the steps in any of the above embodiments of the model training method based on secure multi-party computation.

[0087] The above scheme inputs the sample data matching the current node into the initial model corresponding to the current node for model training, resulting in the trained model corresponding to the current node. Since the initial models corresponding to each node are the same, the model update parameters for updating the initial model corresponding to the current node can be determined based on the model parameters of the trained model corresponding to the current node and the trained model parameters corresponding to other nodes. The initial model corresponding to the current node is then updated based on the model update parameters. This avoids disclosing the sample data and neural network models of each node, thereby improving the training efficiency of the model training process while ensuring data security.

[0088] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0089] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0091] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A model training method based on secure multi-party computation, characterized in that, The method is applied to nodes in a secure multi-party computation system, and the method includes: Obtain the initial model corresponding to the current node. The initial models corresponding to all nodes in the secure multi-party computation system are the same. The step of obtaining the initial model corresponding to the current node includes: building a model based on preset model building data to obtain a basic model; initializing the basic model according to the model building data to obtain an initialized basic model; and synchronizing the initialized basic model corresponding to the current node with the initialized basic models corresponding to other nodes to obtain the initial model corresponding to the current node. Select a subset of sample data corresponding to the current node from the preset sample dataset; The step of selecting a subset of sample data corresponding to the current node from a preset sample dataset includes: segmenting the sample dataset according to the node features of each node to obtain each subset of sample data; and selecting the subset of sample data corresponding to the current node from each subset of sample data. The step of selecting the sample data subset corresponding to the current node from each sample data subset includes: performing sampling with replacement from each sample data subset according to the preset weak classifier in the current node and the current model weight distribution of the current node to obtain the sample data subset corresponding to the current node. If there are identical sample data between the sample data subset corresponding to the current node and the sample data subset corresponding to other nodes, then the data in the sample data subset corresponding to the current node or the sample data subset corresponding to other nodes are deduplicated; the data in the deduplicated sample data subset is used as the sample data matching the current node. The obtained sample data matching the current node is input into the initial model corresponding to the current node for model training to obtain the trained model; The model training includes: obtaining gradient update parameters during the model training process based on the sample data matched by the current node; calculating the average value of the gradient update parameters among the nodes, and using the average value as the average gradient update parameter; updating the model parameters of the initial model corresponding to the current node with the average gradient update parameter to obtain the trained model and the model parameters of the trained model. The step of updating the model parameters of the initial model corresponding to the current node with the average gradient update parameters to obtain the trained model and the model parameters of the trained model includes: updating the model parameters of the initial model corresponding to the current node with the average gradient update parameters to obtain the trained model and the updated model parameters; and performing column compression, row compression, or diagonal compression on the updated model parameters to obtain the model parameters of the trained model. Based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes, the model update parameters corresponding to the current node are determined. The secure multi-party computation system includes the current node and other nodes. The step of determining the model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes includes: encrypting the model parameters in the trained model using the key corresponding to the current node based on a preset Secure Boosting method to obtain encrypted model parameters; merging the encrypted model parameters corresponding to each node to obtain aggregate parameters; sending the aggregate parameters to each node; decrypting the model update parameters corresponding to the current node from the aggregate parameters according to the key corresponding to the current node, and calculating the corresponding error rate; The initial model corresponding to the current node is updated according to the model update parameters corresponding to the current node; the step of updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node includes: determining whether the error rate meets the preset evaluation conditions; if not, performing iterative model training; if yes, updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node.

2. The method according to claim 1, characterized in that, The step of determining the model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes, wherein the secure multi-party computation system includes the current node and other nodes, includes: The model parameters in the trained model corresponding to the current node are aggregated with the model parameters in the trained models corresponding to other nodes to obtain aggregated parameters. The model update parameters corresponding to the current node are decrypted from the aggregate parameters based on the key corresponding to the current node.

3. The method according to claim 1, characterized in that, After the step of updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node, the method further includes: Perform model evaluation on the updated model corresponding to the current node; If the updated model meets the preset evaluation conditions, then model training is paused; If the updated model does not meet the preset evaluation conditions, then model training continues.

4. A model training device based on secure multi-party computation, characterized in that, include: The acquisition module is used to acquire the initial model corresponding to the current node. The initial models corresponding to each node participating in the secure multi-party computation are the same. The step of obtaining the initial model corresponding to the current node includes: building a model based on preset model building data to obtain a basic model; initializing the basic model according to the model building data to obtain an initialized basic model; and synchronizing the initialized basic model corresponding to the current node with the initialized basic models corresponding to other nodes to obtain the initial model corresponding to the current node. The step of selecting a subset of sample data corresponding to the current node from a preset sample dataset includes: segmenting the sample dataset according to the node features of each node to obtain each subset of sample data; selecting the subset of sample data corresponding to the current node from each subset of sample data; the step of selecting the subset of sample data corresponding to the current node from each subset of sample data includes: performing sampling with replacement from each subset of sample data according to the preset weak classifier in the current node and the current model weight distribution of the current node to obtain the subset of sample data corresponding to the current node. If there are identical sample data between the sample data subset corresponding to the current node and the sample data subset corresponding to other nodes, then the data in the sample data subset corresponding to the current node or the sample data subset corresponding to other nodes are deduplicated; the data in the deduplicated sample data subset is used as the sample data matching the current node. A model training module is used to input the acquired sample data matching the current node into the initial model corresponding to the current node for model training to obtain a trained model. The model training includes: obtaining gradient update parameters during model training based on the sample data matching the current node; calculating the average value of the gradient update parameters among the nodes and using the average value as the average gradient update parameter; updating the model parameters of the initial model corresponding to the current node with the average gradient update parameter to obtain the trained model and the model parameters of the trained model; the step of updating the model parameters of the initial model corresponding to the current node with the average gradient update parameter to obtain the trained model and the model parameters of the trained model includes: updating the model parameters of the initial model corresponding to the current node with the average gradient update parameter to obtain the trained model and the updated model parameters; and performing column compression, row compression, or diagonal compression on the updated model parameters to obtain the model parameters of the trained model. The computation module is used to determine the model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes. The nodes participating in the secure multi-party computation include the current node and the other nodes. The step of determining the model update parameters corresponding to the current node based on the model parameters in the trained model corresponding to the current node and the model parameters in the trained models corresponding to other nodes includes: encrypting the model parameters in the trained model using the key corresponding to the current node based on a preset Secure Boosting method to obtain encrypted model parameters; merging the encrypted model parameters corresponding to each node to obtain aggregate parameters; sending the aggregate parameters to each node; decrypting the model update parameters corresponding to the current node from the aggregate parameters according to the key corresponding to the current node, and calculating the corresponding error rate. An update module is used to update the initial model corresponding to the current node according to the model update parameters corresponding to the current node; the step of updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node includes: determining whether the error rate meets the preset evaluation conditions; if not, performing iterative model training; if yes, updating the initial model corresponding to the current node according to the model update parameters corresponding to the current node.

5. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 3.

Images