Mask-based global model training method, device and equipment

Abstract

The invention relates to the field of privacy protection, and provides a mask-based global model training method, device and equipment, and the method comprises the steps: a server initializes model parameters and various masks, and a client interacts with the server to obtain a global mask and update a local mask. And the client processes model parameters by using the updated mask to obtain model output, and updates the mask according to the model output. Then, binary processing and random flipping are carried out on the masks, and then the masks are uploaded to a server; and the server selects a client set, aggregates and corrects the mask uploaded by the client set, and generates a new global mask for next iteration. And iterating for multiple times until a preset number of times is met, and finally outputting a trained global model. According to the method, the user privacy is protected through the mask technology, the global model training is realized, and the privacy security of model training is improved.

Description

Technical Field

[0001] The present invention relates to the field of privacy protection technology, and in particular to a mask-based global model training method, device and equipment. Background Art

[0002] In recent years, with the rapid development of big data and artificial intelligence technology, deep learning has made significant progress in many fields such as image recognition, speech recognition, and natural language processing. In order to improve the training speed and data processing capabilities of deep learning, distributed learning methods have emerged. By distributing huge data sets and computing tasks to multiple machines for parallel processing, distributed learning effectively solves the bottleneck problem of single-machine training in data storage and computing resources, greatly improving the efficiency and scalability of model training.

[0003] Among the many implementations of distributed learning, Federated Learning (FL) has attracted much attention due to its unique privacy protection mechanism. Federated learning allows participants to train models locally and complete the aggregation of global models by sharing model weights or gradient updates with a central server, thereby avoiding the direct transmission of original data and reducing the risk of data leakage. However, although federated learning has shown certain advantages in protecting the privacy of original data, studies have shown that it is still possible for the server to extract the client's private information from the shared model weights or gradients. The risk of this privacy leakage mainly comes from two types of privacy inference attacks: data reconstruction attacks and attribute inference attacks. The former aims to reconstruct the client's original training samples, while the latter focuses on inferring sensitive attributes in the training set, such as the user's gender, age, etc.

[0004] In order to deal with these privacy leakage risks, academia and industry have explored a variety of privacy protection technologies, mainly including encryption, differential privacy, trusted execution environments, and defense strategies against specific attacks. Encryption-based methods protect confidentiality during data exchange through homomorphic encryption or multi-party computing, but this method is often accompanied by high computational and communication overhead. Differential privacy protects privacy by adding noise to model updates. Although it can effectively resist data reconstruction attacks, it may significantly reduce the accuracy of the model. Trusted execution environments (such as Intel SGX) protect the confidentiality of code and data by isolating the execution environment, but still face challenges such as memory limitations in actual deployment. In addition, there are some specially designed defense methods aimed at resisting specific data reconstruction attacks, but these methods generally cannot fully protect sensitive attribute information. Summary of the invention

[0005] The present invention provides a mask-based global model training method, device and equipment, which solves the defect of data privacy leakage in the prior art and realizes safe, efficient and fast global model training.

[0006] The present invention provides a mask-based global model training method, comprising the following steps: Step 1: The server randomly initializes model parameters, generates a global mask of the server and a local mask of the client based on the model parameters, and determines a random flip probability vector according to the local mask; Step 2: The client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask; Step 3: The client applies the second local mask to the model parameters to obtain a model output, obtains a gradient of the second local mask according to the model output, the training set, the second local mask, and the model parameters, and updates the second local mask according to the gradient of the second local mask to obtain a third local mask; Step 4: The client performs binary processing on the third local mask to obtain a fourth local mask; randomly flips the fourth local mask according to the random flip probability vector to obtain a fifth local mask; the flipping is to invert the binary value; Step 5, each client uploads the local fifth local mask to the server; Step 6: The server randomly selects one or more clients from all clients to form a client set, aggregates and corrects the fifth local mask uploaded by each client in the client set to obtain a sixth local mask; and uses the sixth local mask as a global mask for interaction with the client in the next iteration; Iterate steps 2 to 6 until the number of iterations meets the preset requirement, then terminate the training and output the trained global model.

[0007] According to a mask-based global model training method provided by the present invention, the model parameters are randomly initialized, a global mask of the server and a local mask of the client are generated based on the model parameters, and a random flipping probability vector is determined according to the number of model layers. Specifically, the method includes: after the server determines the global model, the model parameters of the global model are randomly initialized and fixed by the server; the server randomly initializes a global mask with the same shape as the model parameters; each client randomly initializes a local mask with the same shape as the model parameters with the same initial value; and a random flipping probability vector is determined according to the number of layers of the global model and the local mask.

[0008] According to a mask-based global model training method provided by the present invention, the client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask, specifically including: the client obtains the global mask of this iteration from the server; performs absolute value operation on each corresponding element in the first local mask matrix; performs positive and negative processing on the result of the absolute value operation according to the value of each element in the global mask; replaces each element in the first local mask with the value after positive and negative processing of each element, to obtain an updated second mask matrix.

[0009] According to a mask-based global model training method provided by the present invention, the second local mask is applied to the model parameters to obtain the model output, and the gradient of the second local mask is obtained according to the model output, the training set, the second local mask, and the model parameters, specifically including: calculating the model output according to the second local mask and the model parameters; determining the gradient matrix of the activation function according to the second local mask; determining the loss function according to the training set and the model output; determining the gradient of the second local mask according to the model output, the training set, the gradient matrix and the loss function.

[0010] According to a mask-based global model training method provided by the present invention, the fourth local mask is randomly flipped according to the random flip probability vector to obtain a fifth local mask, which specifically includes: determining the random flip probability of each layer in the random flip probability vector; flipping the element value of the layer corresponding to the fourth local mask according to the random flip probability of each layer, and obtaining the fifth local mask after the flipping is completed.

[0011] According to a mask-based global model training method provided by the present invention, the second local mask is updated according to the gradient of the second local mask to obtain a third local mask, which specifically includes: determining a temporary amount according to the gradient and learning rate of the second local mask; obtaining the difference between the second local mask and the temporary amount as the third local mask.

[0012] The present invention also provides a mask-based global model training device, comprising the following modules: An initialization module, used for randomly initializing model parameters on the server side, generating a global mask of the server and a local mask of the client based on the model parameters, and determining a random flip probability vector according to the local mask; A local mask acquisition module, used for the client to interact with the server to acquire the global mask of this iteration, and update the first local mask of this iteration to a second local mask according to the global mask; A local mask training module, used for the client to apply the second local mask to the model parameters to obtain a model output, obtain the gradient of the second local mask according to the model output, the training set, the second local mask, and the model parameters, and update the second local mask according to the gradient of the second local mask to obtain a third local mask; A local mask random flip module, used for the client to perform binary processing on the third local mask to obtain a fourth local mask; according to the random flip probability vector, randomly flip the fourth local mask to obtain a fifth local mask; the flipping is to invert the binary value; A local mask transmission module, used for each client to upload the local fifth local mask to the server; The online aggregation module is used for the server to randomly select one or more clients from all clients to form a client set, aggregate and correct the fifth local mask uploaded by each client in the client set to obtain a sixth local mask; and use the sixth local mask as a global mask for interaction with the client in the next iteration; The model output module is used to control the iterative execution of the local mask acquisition module, the local mask training module, the local mask random flipping module, the local mask transmission module and the online aggregation module until the number of iterations meets the preset requirements, then terminate the training and output the trained global model.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the mask-based global model training method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the mask-based global model training methods described above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the mask-based global model training methods described above.

[0016] The present invention provides a mask-based global model training method, device and equipment, including the following beneficial effects: by training based on masks, the model parameters used for communication are converted into binary masks, which can resist data reconstruction attacks and reduce communication overhead compared with floating-point parameters, thereby helping to reduce the risk of data leakage, thereby enhancing the protection of data privacy. Since masks are transmitted instead of complete model parameters or a large amount of original data, this training method can effectively reduce the communication cost between clients and servers in a distributed learning environment. Masks are usually much smaller than original data or complete model parameters, thereby saving bandwidth and transmission time. A random flip probability vector is introduced during the training process to randomly flip the mask to protect sensitive attribute information and resist attribute reasoning attacks. At the same time, the probability of flipping the mask is determined according to the attribute information embedded in each layer, taking into account the accuracy of the model. The present invention is suitable for a distributed learning environment, in which multiple clients can be trained in parallel, and the server is responsible for aggregating and updating the mask. This architecture has good scalability and can easily increase or decrease the number of clients participating in the training to adapt to different computing resources and data distribution conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the mask-based global model training method provided by the present invention.

[0019] Figure 2 It is a flow chart of initializing model parameters, global mask, local mask and random flip probability vector provided by the present invention.

[0020] Figure 3 This is the second flowchart of the mask-based global model training method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the mask-based global model training device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In recent years, deep learning has been widely used in various fields such as image, speech, and natural language processing. With the rise of the concept of big data, people's demand for deep learning training speed and data processing capabilities has continued to increase. The traditional single-machine-based training process is difficult to meet the huge data storage and computing requirements. In order to increase the speed of data computing, reduce the time consumption of tasks, and improve scalability, tasks with huge data and computing volume need to be distributed and deployed on multiple machines. For this reason, a distributed deep learning training method has been proposed.

[0025] However, since the original data often contains a lot of sensitive information, the information interaction between the various participants in distributed learning is at risk of privacy leakage. Federated Learning (FL), as a distributed deep learning method, avoids revealing sensitive information by sharing model weights or gradient updates with a central server, thereby protecting the privacy of the original data. However, researchers have shown that an honest server can obtain private information from shared model weights or gradients. Specifically, many privacy inference attacks have been proposed, mainly including data reconstruction attacks and attribute inference attacks. The former attempts to reconstruct the original training samples of the client, while the latter does not focus on specific data and aims to infer sensitive attributes in the training set, such as gender. Therefore, in the federated learning scenario, how to defend against different attacks while maintaining good accuracy is an urgent problem to be solved.

[0026] In order to solve the above privacy issues, a variety of protection technologies have been proposed, which can be mainly divided into the following four types.

[0027] 1.Secure aggregation method based on encryption.

[0028] 2. Differential Privacy (DP) mechanism.

[0029] 3. A privacy-preserving federated learning framework using a trusted execution environment (TEE).

[0030] 4. Defense methods specifically designed to defend against data reconstruction attacks.

[0031] However, the above-mentioned existing distributed learning privacy protection schemes are difficult to strike a good balance between accuracy, privacy and efficiency.

[0032] Secure aggregation ensures that the central server can only obtain locally updated aggregation results by utilizing homomorphic encryption and secret sharing techniques, so it inevitably brings considerable additional computational and communication overheads.

[0033] Differential privacy mechanisms add Gaussian or Laplace noise to model updates, providing theoretical upper bounds on membership, which refers to whether a sample is included in the training set or whether the client participates in global updates. However, it leads to significant accuracy degradation in order to provide sufficient privacy protection against data reconstruction attacks and attribute inference attacks.

[0034] Other defense methods specifically designed to defend against data reconstruction attacks are ineffective for protecting attribute information.

[0035] In general, in distributed systems, existing defense measures cannot achieve complete defense against the leakage of raw data and sensitive attribute information without causing significant utility degradation or large defense overhead.

[0036] The purpose of this invention is to design a new privacy protection framework for distributed learning scenarios, which can balance accuracy, privacy and communication efficiency.

[0037] The present invention improves the model parameters transmitted during communication. Through mask-based training, the model parameters used for communication are converted into binary masks. Compared with floating-point parameters, they can resist data reconstruction attacks and reduce communication overhead. In addition, the present invention designs a privacy-aware random mechanism to randomly flip the mask to protect sensitive attribute information and resist attribute reasoning attacks. At the same time, the probability of flipping the mask is determined according to the attribute information embedded in each layer, taking into account the accuracy of the model.

[0038] Combine the following Figure 1-Figure 5 Embodiments of the present invention are described in detail.

[0039] The privacy protection method for distributed learning provided by the present invention takes federated learning as an example and is implemented by the client and server participating in federated learning. The method mainly includes six steps: initialization, mask acquisition, mask training, random mask flipping, mask upload, and online model aggregation.

[0040] Figure 1 is one of the flowcharts of the mask-based global model training method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: S110, the server randomly initializes model parameters, generates a global mask of the server and a local mask of the client based on the model parameters, and determines a random flip probability vector according to the local mask.

[0041] According to a mask-based global model training method provided by the present invention, the model parameters are randomly initialized, a global mask of the server and a local mask of the client are generated based on the model parameters, and a random flip probability vector is determined according to the number of model layers, specifically including: after the server determines the global model, the model parameters of the global model are randomly initialized and fixed by the server; the server randomly initializes a global mask with the same shape as the model parameters; each client randomly initializes a local mask with the same shape as the model parameters with the same initial value; and a random flip probability vector is determined according to the number of layers of the global model and the local mask. The global mask and the local mask are collectively referred to as masks.

[0042] Specifically, step 1, initialization, includes fixing model parameters, generating global masks, generating local masks, and determining random flip probability vectors. The specific steps are as follows: Figure 2 As shown, it includes the following four steps: Step 11: The server randomly initializes the parameters of the model and keeps them fixed. After the server determines the global model, it randomly initializes the parameters of the model and keeps them fixed; Step 12: The server randomly initializes the global binary mask. Then the server randomly initializes the global binary mask , whose shape is the same as the model parameters, and each element is randomly 0 or 1; Step 13: All clients use the same seed to randomly initialize the real number mask used locally , whose shape is the same as the model parameters; the same seed means the same starting point.

[0043] Step 14: The server determines the random flip probability vector , where L is the number of layers of the model. Each element corresponds to each layer of the model parameters and each layer of the mask, indicating that Layer mask with The probability of The probability of random flipping is , and the value of each element is determined by the attribute leakage risk of the corresponding layer of the model.

[0044] Jordi The probability of random flipping of a layer mask is , generate auxiliary data with similar distribution to the training set , depending on whether it contains The attributes in , two models are trained based on the mask, denoted as and , respectively take the Layer Mask and ,but The calculation method is:

[0045] Among them, cos( ) indicates the calculation of cosine similarity. The initial random flip probability specified by the client and the server, .

[0046] By initializing the global mask and the local mask and combining them with the random flip probability vector, the data can be effectively masked during the training process, thereby protecting the privacy of the original data. This masking mechanism can prevent data leakage, which is particularly important when processing sensitive or private data.

[0047] S120: The client interacts with the server to obtain a global mask of this iteration, and updates a first local mask of this iteration to a second local mask according to the global mask.

[0048] According to a mask-based global model training method provided by the present invention, the client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask, specifically including: the client obtains the global mask of this iteration from the server; performs absolute value operation on each corresponding element in the first local mask matrix; performs positive and negative processing on the result of the absolute value operation according to the value of each element in the global mask; replaces each element in the first local mask with the value after positive and negative processing of each element, to obtain an updated second mask matrix.

[0049] Specifically, in step 2, obtaining a mask, the client obtains a global mask from the server, the global mask being in binary form, and updates the first local mask based on the global mask.

[0050] Specifically, the client obtains the first Global binary mask of interactions , and update the local real number mask using the following formula :

[0051] Among them, abs( ) means taking the absolute value, represents the first Line Column elements, represents the local real mask matrix Line It should be noted that the first local mask is obtained before the absolute value operation, and the second local mask is obtained after the absolute value is taken and the positive and negative operations are performed. Both the first local mask and the second local mask are in real number form.

[0052] The introduction of the global mask adds a layer of shielding to the original data, further enhancing the protection of data privacy. Since the global mask is in binary form, it provides an additional obfuscation mechanism when combined with the local real number mask, making the original data more difficult to reverse parse.

[0053] S130. The client applies the second local mask to the model parameters to obtain a model output, obtains a gradient of the second local mask according to the model output, the training set, the second local mask, and the model parameters, and updates the second local mask according to the gradient of the second local mask to obtain a third local mask.

[0054] According to a mask-based global model training method provided by the present invention, the second local mask is applied to the model parameters to obtain the model output, and the gradient of the second local mask is obtained according to the model output, the training set, the second local mask, and the model parameters, specifically including: calculating the model output according to the second local mask and the model parameters; determining the gradient matrix of the activation function according to the second local mask; determining the loss function according to the training set and the model output; determining the gradient of the second local mask according to the model output, the training set, the gradient matrix and the loss function.

[0055] According to a mask-based global model training method provided by the present invention, the second local mask is updated according to the gradient of the second local mask to obtain a third local mask, which specifically includes: determining a temporary amount according to the gradient and learning rate of the second local mask; obtaining the difference between the second local mask and the temporary amount as the third local mask.

[0056] Specifically, in step 3, training mask, the client applies the second local mask to the model parameters, and then uses the training set, the second local mask, and the model parameters to obtain a gradient to obtain a new third local mask.

[0057] Specifically, the client updates the second local mask Train the local model. For each sample in the training set , model parameters , the model is calculated as:

[0058] in represents the Hadamard product, which is the product of the elements in the same position of the matrix. Represents the Sigmoid function, that is , The threshold is The hard threshold function is that for each element of the matrix, if the value is greater than , it becomes 1 if the value is less than or equal to , it becomes 0. In this example, .

[0059] This gives the second local mask The gradient of is:

[0060] in is the loss function of the model, is the gradient matrix of the Sigmoid function, Line Column element value , Indicates that the hard threshold function The calculated matrix Line The element value of the column.

[0061] Update the second local mask in real form by The third local mask in real form :

[0062] in is the learning rate.

[0063] By training masks instead of directly training model parameters, the original data is better protected. This is because the masking mechanism plays a role in data shielding during the training process, reducing the risk of data leakage. In a distributed learning environment, the client and server need to frequently exchange information. Since masks are transmitted instead of complete model parameters or original data, this greatly reduces the communication cost.

[0064] S140. The client performs binary processing on the third local mask to obtain a fourth local mask; randomly flips the fourth local mask according to the random flip probability vector to obtain a fifth local mask; the flipping is to invert the binary value.

[0065] According to a mask-based global model training method provided by the present invention, the fourth local mask is randomly flipped according to the random flip probability vector to obtain a fifth local mask, which specifically includes: determining the random flip probability of each layer in the random flip probability vector; flipping the element value of the layer corresponding to the fourth local mask according to the random flip probability of each layer, and obtaining the fifth local mask after the flipping is completed.

[0066] Specifically, in step 4, mask randomization, the client randomly flips the local mask according to the random flip probability vector, where flipping refers to inverting the value of the binary mask.

[0067] Specifically, Client count Fourth local mask for the interaction , that is, the third local mask in real form is processed by the activation function to obtain the fourth local mask in binary form. Then for the Layer 4 local mask, with The probability of randomly flipping The value of each element of the fourth local mask of the layer, where flipping means inverting the binary value. After each element is randomly flipped, the fifth local mask in binary form is obtained. .

[0068] By randomly flipping the value of the mask, the information of the original data is further obfuscated, making it more difficult to reverse-parse the original data from the mask. This randomness increases the difficulty for attackers to try to recover sensitive information, thereby improving the level of privacy protection.

[0069] S150: Each client uploads the local fifth local mask to the server.

[0070] Specifically, step 5, upload mask. Each client uploads its own binary fifth local mask Upload to the server.

[0071] S160. The server randomly selects one or more clients from all clients to form a client set, aggregates and corrects the fifth local mask uploaded by each client in the client set to obtain a sixth local mask; and uses the sixth local mask as a global mask for interaction with the client in the next iteration.

[0072] Specifically, in step 6, online model aggregation, the server aggregates the collected fifth local masks, and then eliminates the influence of random flipping to obtain a corrected local mask as the global mask used for the next interaction with the client.

[0073] Specifically, the server receives Randomly select from clients, forming a collection , upload them to the fifth local mask in binary form, the element at row j, column i Perform aggregation to obtain the elements in the sixth local mask , the calculation formula is:

[0074] Among them, for the The binary mask of each layer is used to eliminate the influence of random flipping layer by layer through the following formula:

[0075] The corrected mask is used as the global mask for the next interaction with the client. .

[0076] S170, iteratively execute S120-S160 until the number of iterations meets the preset requirement, then terminate the training and output the trained global model.

[0077] Specifically, execute steps 2 to 6 again until the number of interactions meets the preset total number. , the training is terminated and the trained global model is output.

[0078] By aggregating the local masks of multiple clients and eliminating the impact of random flipping, the server can obtain a more accurate global mask. Since the local masks of the clients are randomly flipped before uploading, this protects the privacy of the client data to a certain extent. At the same time, the process of aggregating and eliminating the impact of random flipping on the server side is equivalent to "averaging" multiple masks, which helps reduce the impact of individual outliers or noise on the global model, thereby enhancing the robustness of the model.

[0079] like Figure 3 The figure shows the second schematic diagram of a mask-based model training process provided by the present invention. The global model is initialized through the initialization module, the global binary mask, model parameters and random flip vector are obtained, the global binary mask is obtained through the mask acquisition module, and the updated local real number mask is output; the training mask module is used to train according to the model parameters, training set and updated local real number mask; the trained local real number mask is input into the random flip mask module for binary conversion, and it is flipped in combination with the random flip probability vector, and the local binary mask after random flipping is uploaded; the trained model is output through the online model aggregation module.

[0080] In summary, the present invention proposes a model training method based on binary mask. By using mask-based training, the communication cost between the client and the server in distributed learning can be compressed, the communication efficiency can be improved, and the privacy protection effect against data reconstruction attacks can be provided, thereby achieving defense against data reconstruction attacks.

[0081] This paper proposes a layer-by-layer parameter randomization method, which is used to determine the probability of mask randomization by quantifying the attribute leakage risk of each layer of the model mask. Using mask randomization and adjusting the probability of randomization, this method can achieve defense against attribute inference attacks, provide more protection for sensitive masks, and reduce the disturbance added to masks with lower privacy risks, thereby alleviating the problem of model accuracy decline caused by defense measures, alleviating the decline in model practicality, and balancing model privacy and practicality.

[0082] The mask-based global model training device provided by the present invention is described below. The mask-based global model training device described below and the mask-based global model training method described above can be referenced to each other.

[0083] like Figure 4 The present invention provides a mask-based global model training device, comprising: Initialization module 410, used for the server to randomly initialize model parameters, generate a global mask of the server and a local mask of the client based on the model parameters, and determine a random flip probability vector according to the local mask; The local mask acquisition module 420 is used for the client to interact with the server to acquire the global mask of this iteration, and update the first local mask of this iteration to the second local mask according to the global mask; A local mask training module 430, configured for the client to apply the second local mask to model parameters to obtain a model output, obtain a gradient of the second local mask according to the model output, a training set, the second local mask, and the model parameters, and update the second local mask according to the gradient of the second local mask to obtain a third local mask; A local mask random flip module 440 is used for the client to perform binary processing on the third local mask to obtain a fourth local mask; according to the random flip probability vector, the fourth local mask is randomly flipped to obtain a fifth local mask; the flipping is to invert the binary value; A local mask transmission module 450, configured for each client to upload a local fifth local mask to the server; The online aggregation module 460 is used for the server to randomly select one or more clients from all clients to form a client set, aggregate and correct the fifth local mask uploaded by each client in the client set to obtain a sixth local mask; and use the sixth local mask as a global mask for interaction with the client in the next iteration; The model output module 470 is used to control the iterative execution of the local mask acquisition module, the local mask training module, the local mask random flipping module, the local mask transmission module and the online aggregation module until the number of iterations meets the preset requirements, then terminate the training and output the trained global model.

[0084] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the mask-based global model training method, which includes: step 1, the server randomly initializes the model parameters, generates the global mask of the server and the local mask of the client based on the model parameters, and determines the random flip probability vector according to the local mask; step 2, the client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask; step 3, the client applies the second local mask to the model parameters to obtain the model output, obtains the gradient of the second local mask according to the model output, the training set, the second local mask and the model parameters, and updates the second local mask according to the gradient of the second local mask to obtain the third local mask. Mask; Step 4, the client performs binary processing on the third local mask to obtain a fourth local mask; according to the random flip probability vector, the fourth local mask is randomly flipped to obtain a fifth local mask; flipping is to invert the binary value; Step 5, each client uploads the local fifth local mask to the server; Step 6, the server randomly selects one or more clients from all clients to form a client set, aggregates and corrects the fifth local mask uploaded by each client in the client set, and obtains a sixth local mask; The sixth local mask is used as a global mask for interaction with the client in the next iteration; Iterate steps 2-6 until the number of iterations meets the preset requirements, then terminate the training and output the trained global model.

[0085] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the mask-based global model training method provided by the above methods, and the method includes: step 1, the server randomly initializes the model parameters, generates the global mask of the server and the local mask of the client based on the model parameters, and determines the random flipping probability vector according to the local mask; step 2, the client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask; step 3, the client applies the second local mask to the model parameters to obtain the model output, and obtains the second local mask according to the model output, the training set, the second local mask, and the model parameters. The gradient of the code is obtained by updating the second local mask according to the gradient of the second local mask to obtain a third local mask; step 4, the client performs binary processing on the third local mask to obtain a fourth local mask; according to the random flip probability vector, the fourth local mask is randomly flipped to obtain a fifth local mask; flipping is to invert the binary value; step 5, each client uploads the local fifth local mask to the server; step 6, the server randomly selects one or more clients from all clients to form a client set, aggregates and corrects the fifth local mask uploaded by each client in the client set, and obtains a sixth local mask; the sixth local mask is used as a global mask for interaction with the client in the next iteration; iteratively execute steps 2-6 until the number of iterations meets the preset requirements, then terminate the training and output the trained global model.

[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the mask-based global model training method provided by the above-mentioned methods, the method comprising: step 1, the server randomly initializes the model parameters, generates the global mask of the server and the local mask of the client based on the model parameters, and determines the random flipping probability vector according to the local mask; step 2, the client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask; step 3, the client applies the second local mask to the model parameters to obtain the model output, obtains the gradient of the second local mask according to the model output, the training set, the second local mask, and the model parameters, and obtains the gradient of the second local mask according to the second local mask. The gradient updates the second local mask to obtain the third local mask; step 4, the client performs binary processing on the third local mask to obtain the fourth local mask; according to the random flip probability vector, the fourth local mask is randomly flipped to obtain the fifth local mask; flipping is to invert the binary value; step 5, each client uploads the local fifth local mask to the server; step 6, the server randomly selects one or more clients from all clients to form a client set, aggregates and corrects the fifth local mask uploaded by each client in the client set, and obtains the sixth local mask; the sixth local mask is used as the global mask for interaction with the client in the next iteration; iteratively execute steps 2-6 until the number of iterations meets the preset requirements, then terminate the training and output the trained global model.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mask-based global model training method, characterized in that: include: Step 1: The server randomly initializes model parameters, generates a global mask of the server and a local mask of the client based on the model parameters, and determines a random flip probability vector according to the local mask; Step 2: The client interacts with the server to obtain the global mask of this iteration, and updates the first local mask of this iteration to the second local mask according to the global mask; Step 3: The client applies the second local mask to the model parameters to obtain a model output, obtains a gradient of the second local mask according to the model output, the training set, the second local mask, and the model parameters, and updates the second local mask according to the gradient of the second local mask to obtain a third local mask; Step 4: The client performs binary processing on the third local mask to obtain a fourth local mask; According to the random flip probability vector, randomly flip the fourth local mask to obtain a fifth local mask; the flipping is to invert the binary value; Step 5, each client uploads the local fifth local mask to the server; Step 6: The server randomly selects one or more clients from all the clients to form a client set, aggregates and corrects the fifth local mask uploaded by each client in the client set, and obtains a sixth local mask; Using the sixth local mask as a global mask for interaction with the client in the next iteration; Iterate steps 2 to 6 until the number of iterations meets the preset requirement, then terminate the training and output the trained global model.

2. The mask-based global model training method according to claim 1, characterized in that: The randomly initialized model parameters, generating a global mask of the server and a local mask of the client based on the model parameters, and determining a random flip probability vector according to the number of model layers, specifically includes: After the server determines the global model, the server randomly initializes the model parameters of the global model and keeps them unchanged; The server randomly initializes a global mask with the same shape as the model parameters; Each client randomly initializes a local mask with the same shape as the model parameters with the same initial value; A random flip probability vector is determined according to the number of layers of the global model and the local mask.

3. The mask-based global model training method according to claim 1, characterized in that: The client interacts with the server to obtain a global mask of this iteration, and updates a first local mask of this iteration to a second local mask according to the global mask, specifically including: The client obtains the global mask for this iteration from the server; Performing an absolute value operation on each corresponding element in the first local mask matrix; Performing positive and negative processing on the result of the absolute value operation according to the value of each element in the global mask; Each element in the first local mask is replaced by a value after positive and negative processing of each element, to obtain an updated second mask matrix.

4. The mask-based global model training method according to claim 1, characterized in that: The applying the second local mask to the model parameters to obtain a model output, and obtaining a gradient of the second local mask according to the model output, a training set, the second local mask, and the model parameters specifically includes: calculating a model output based on the second local mask and the model parameters; Determining a gradient matrix of an activation function according to the second local mask; Determining a loss function based on the training set and the model output; The gradient of the second local mask is determined based on the model output, the training set, the gradient matrix, and the loss function.

5. The mask-based global model training method according to claim 1, characterized in that: The randomly flipping the fourth local mask according to the random flip probability vector to obtain a fifth local mask specifically includes: Determine the random flip probability of each layer in the random flip probability vector; The element value of the layer corresponding to the fourth local mask is flipped according to the random flipping probability of each layer, and the fifth local mask is obtained after the flipping is completed.

6. The mask-based global model training method according to claim 1, characterized in that: The updating of the second local mask according to the gradient of the second local mask to obtain a third local mask specifically includes: determining a temporary amount according to a gradient of the second local mask and a learning rate; A difference between the second local mask and the temporary amount is obtained as a third local mask.

7. A mask-based global model training device, characterized in that: include: An initialization module, used for randomly initializing model parameters on the server side, generating a global mask of the server and a local mask of the client based on the model parameters, and determining a random flip probability vector according to the local mask; A local mask acquisition module, used for the client to interact with the server to acquire the global mask of this iteration, and update the first local mask of this iteration to a second local mask according to the global mask; A local mask training module, used for the client to apply the second local mask to the model parameters to obtain a model output, obtain the gradient of the second local mask according to the model output, the training set, the second local mask, and the model parameters, and update the second local mask according to the gradient of the second local mask to obtain a third local mask; A local mask random flipping module, used for the client to perform binary processing on the third local mask to obtain a fourth local mask; According to the random flip probability vector, randomly flip the fourth local mask to obtain a fifth local mask; the flipping is to invert the binary value; A local mask transmission module, used for each client to upload the local fifth local mask to the server; An online aggregation module, configured for the server to randomly select one or more clients from all clients to form a client set, aggregate and correct the fifth local mask uploaded by each client in the client set, and obtain a sixth local mask; Using the sixth local mask as a global mask for interaction with the client in the next iteration; The model output module is used to control the iterative execution of the local mask acquisition module, the local mask training module, the local mask random flipping module, the local mask transmission module and the online aggregation module until the number of iterations meets the preset requirements, then terminate the training and output the trained global model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the mask-based global model training method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mask-based global model training method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the mask-based global model training method as described in any one of claims 1 to 6 is implemented.

Images