Training method, device and electronic equipment for federated neural network

Abstract

The embodiments of the present application provide a training method, device, electronic device and computer-readable storage medium for a federated neural network, and relate to the field of artificial intelligence. The method includes: determining the reverse partial derivative of the batch of first-party data output by the top-level model, determining the distribution range of the noise data according to the distribution of the reverse partial derivatives of the positive samples and the negative samples and the degree of disturbance; randomly sampling noise data from the distribution range for each reverse partial derivative, adding noise data to the reverse partial derivative, and obtaining the perturbed reverse partial derivative; inputting the perturbed reverse partial derivative into the underlying model, so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative. The embodiments of the present application do not require iteration or long-term complex calculations like the existing technology, and are more efficient than related technologies. When the efficiency is improved, a lower maximum acceptable distance index can be provided, thereby improving the protection capability of the label.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology. Specifically, the present application relates to a training method, device and electronic equipment for a federated neural network. Background Art

[0002] Federated Learning (FL) can leverage multiple data sources to train deep learning models (i.e., artificial neural network models) without requiring data to leave the domain, and also provide model prediction services. While ensuring user privacy and data security, FL leverages multi-party data collaboration to improve the performance of deep learning models.

[0003] Existing vertical federated neural network training methods either have the risk of leaking label information through backward derivatives or are inefficient in order to protect label information. Summary of the Invention

[0004] The present invention provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training a federated neural network, which can address the aforementioned problems of the prior art. The technical solution is as follows:

[0005] According to one aspect of an embodiment of the present application, a federated neural network model is provided, including at least a bottom model of a first participant and a top model of a second participant, and a training method includes:

[0006] During backpropagation, determine the reverse partial derivative of the batch of first-party data output by the top model;

[0007] Determine the positive and negative samples in the first party's data based on the label information set by the second party for the first party's data, and determine the distribution range of the noise data based on the distribution of the reverse partial derivatives of the positive and negative samples and the preset disturbance degree;

[0008] For each reverse partial derivative, randomly sample noise data from the distribution range, add noise data to the reverse partial derivative, and obtain the perturbed reverse partial derivative;

[0009] The perturbed reverse partial derivative is input into the underlying model so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative.

[0010] As an optional embodiment, determining the distribution range of the noise data according to the distribution of the reverse partial derivatives of the positive samples and the negative samples and a preset disturbance degree includes: determining the difference between the first expectation and the second expectation;

[0011] According to the difference and the preset disturbance degree, the covariance matrix of the first participant's data after reverse partial derivative disturbance is obtained;

[0012] Obtaining a covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix, and the second covariance matrix;

[0013] Modeling is performed based on the covariance matrix of the noise data distribution and the preset third expectation to obtain the distribution range of the noise data;

[0014] Among them, the first expectation and the first covariance matrix are the expectation and covariance matrices of the first multivariate Gaussian distribution modeled according to the reverse partial derivative of the positive sample, and the second expectation and the second covariance matrix are the expectation and covariance matrices of the second multivariate Gaussian distribution modeled according to the reverse partial derivative of the negative sample.

[0015] As an optional embodiment, the difference includes sub-differences between the first expectation and the second expectation in each hidden dimension of the underlying model;

[0016] According to the difference and the predetermined disturbance degree, the covariance matrix after disturbance is obtained, including:

[0017] Each sub-difference in the difference is squared and summed up. The covariance matrix after perturbation is obtained based on the summation result, the identity matrix and the degree of perturbation.

[0018] As an optional embodiment, according to the summed result, the identity matrix and the disturbance degree, a covariance matrix after disturbance is obtained, including:

[0019] Calculate the quotient between the summed result and the degree of disturbance, and take the product of the quotient and the identity matrix as the covariance matrix after disturbance.

[0020] As an optional embodiment, obtaining the covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix of the first multivariate Gaussian distribution, and the second covariance matrix of the second multivariate Gaussian distribution includes:

[0021] The differences between the covariance matrix after disturbance and the first and second covariance matrices are calculated respectively to obtain a third covariance matrix for the noise data distribution of the positive samples and a fourth covariance matrix for the noise data distribution of the negative samples.

[0022] As an optional embodiment, modeling is performed based on the covariance matrix of the noise data distribution and a preset expectation to obtain the distribution range of the noise data, including:

[0023] Modeling based on the third covariance matrix and the preset expectation to obtain a first distribution range of the noise data for the positive sample;

[0024] According to the fourth covariance matrix and the preset expectation modeling, a second distribution range of the noise data for the negative sample is obtained.

[0025] As an optional embodiment, adding noise data to the reverse partial derivative to obtain the perturbed reverse partial derivative includes:

[0026] Randomly sample first noise data from a first distribution range, and randomly sample second noise data from a second distribution range;

[0027] Performing weighted summation on the first noise data and the second noise data to obtain disturbance data;

[0028] Add perturbation data to the reverse partial derivative to obtain the perturbed reverse partial derivative.

[0029] As an optional embodiment, performing weighted summation on the first noise data and the second noise data includes:

[0030] If the first participant data corresponding to the reverse partial derivative is a positive sample, the weight corresponding to the first noise data is set to the first value, and the weight corresponding to the second noise data is set to the second value;

[0031] If the first participant data corresponding to the reverse partial derivative is a negative sample, the weight corresponding to the first noise data is set to the second value, and the weight corresponding to the second noise data is set to the first value;

[0032] The first value is greater than the second value.

[0033] As an optional embodiment, the first expectation and the second expectation are both vectors of dimension d; the first covariance matrix and the second covariance matrix are both d×d diagonal matrices; where d is the hidden layer dimension of the underlying model.

[0034] According to another aspect of an embodiment of the present application, a training apparatus for a federated neural network is provided. The federated neural network model includes at least a bottom-level model of a first participant and a top-level model of a second participant. The apparatus includes:

[0035] A data acquisition module is used to determine the reverse partial derivative of the batch of first-party data output by the top-level model;

[0036] A noise range determination module is configured to determine positive and negative samples in the first party's data based on the label information set by the second party for the first party's data, and to determine the distribution range of the noise data based on the distribution of the reverse partial derivatives of the positive and negative samples and a preset disturbance degree;

[0037] A noise adding module is used to randomly sample noise data from a distribution range for each reverse partial derivative, add noise data to the reverse partial derivative, and obtain the perturbed reverse partial derivative;

[0038] A parameter updating module is used to input the perturbed reverse partial derivative into the underlying model so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative;

[0039] As an optional embodiment, the noise range determination module includes:

[0040] a difference calculation submodule, configured to determine a difference between the first expectation and the second expectation;

[0041] A post-perturbation covariance submodule is used to obtain the covariance matrix of the batch of first-party data after reverse partial derivative perturbation based on the difference and a predetermined degree of perturbation;

[0042] A noise covariance submodule, configured to obtain a covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix, and the second covariance matrix;

[0043] The disturbance distribution submodule is used to perform modeling based on the covariance matrix of the noise data distribution and the preset third expectation to obtain the distribution range of the noise data;

[0044] Among them, the first expectation and the first covariance matrix are the expectation and covariance matrices of the first multivariate Gaussian distribution modeled according to the reverse partial derivative of the positive sample, and the second expectation and the second covariance matrix are the expectation and covariance matrices of the second multivariate Gaussian distribution modeled according to the reverse partial derivative of the negative sample.

[0045] As an optional embodiment, the difference includes sub-differences between the first expectation and the second expectation in each hidden dimension of the underlying model;

[0046] The perturbed covariance submodule is specifically used to square each sub-difference in the difference and then sum them up, and obtain the perturbed covariance matrix based on the summed result, the unit matrix and the perturbation degree.

[0047] As an optional embodiment, the post-disturbance covariance submodule is specifically used to calculate the quotient between the summed result and the disturbance degree, and use the product of the quotient and the identity matrix as the post-disturbance covariance matrix.

[0048] As an optional embodiment, the noise covariance submodule is specifically used to: calculate the difference between the perturbed covariance matrix and the first covariance matrix and the second covariance matrix, respectively, to obtain a third covariance matrix for the noise data distribution of positive samples and a fourth covariance matrix for the noise data distribution of negative samples.

[0049] As an optional embodiment, the disturbance distribution submodule includes:

[0050] A positive sample noise distribution unit, configured to obtain a first distribution range of noise data for the positive sample based on a third covariance matrix and a preset expectation model;

[0051] The negative sample noise distribution unit is used to obtain a second distribution range of noise data for negative samples according to the fourth covariance matrix and preset expectation modeling.

[0052] As an optional embodiment, the noise adding module includes:

[0053] A noise data sampling submodule, configured to randomly sample first noise data from a first distribution range and randomly sample second noise data from a second distribution range;

[0054] A weighting submodule, configured to perform weighted summation on the first noise data and the second noise data to obtain disturbance data;

[0055] The disturbance adding submodule is used to add disturbance data to the reverse partial derivative to obtain the perturbed reverse partial derivative.

[0056] As an optional embodiment, the weighting submodule is specifically configured to:

[0057] If the first participant data corresponding to the reverse partial derivative is a positive sample, the weight corresponding to the first noise data is set to the first value, and the weight corresponding to the second noise data is set to the second value;

[0058] If the first participant data corresponding to the reverse partial derivative is a negative sample, the weight corresponding to the first noise data is set to the second value, and the weight corresponding to the second noise data is set to the first value;

[0059] The first value is greater than the second value.

[0060] As an optional embodiment, the first expectation and the second expectation are both vectors of dimension d; the first covariance matrix and the second covariance matrix are both d×d diagonal matrices; where d is the hidden layer dimension of the underlying model.

[0061] According to another aspect of the embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the training method of the federated neural network in the first aspect.

[0062] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the federated neural network of the first aspect are implemented.

[0063] According to one aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the training method of the federated neural network of the first aspect.

[0064] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0065] During the backpropagation process, by determining the reverse partial derivative of the first participant's data output by the top-level model, and based on the label information set by the second participant for the first participant's data, after determining a small batch of positive and negative samples, the distribution range of the noise data can be quickly determined based on the distribution of the reverse partial derivatives of the positive and negative samples and the preset degree of disturbance. Each time the reverse partial derivative is disturbed, noise data can be randomly sampled from the distribution range for scrambling. This does not require iteration or long-term complex calculations like in existing technologies, and is more efficient than related technologies. Furthermore, when the efficiency is improved, a lower maximum acceptable distance indicator can be provided, thereby improving the protection capability of the label. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0067] Figure 1 A flowchart of a vertical federated neural network training process in related technologies;

[0068] Figure 2 A schematic diagram of the architecture of a training system for a federated neural network model provided in an embodiment of the present application;

[0069] Figure 3 A flowchart of a method for training a federated neural network according to an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a process for determining the distribution range of noise data provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a multivariate Gaussian distribution for reverse partial derivative modeling of positive and negative samples provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of a multivariate Gaussian distribution before and after adding perturbations to the reverse partial derivatives of positive and negative samples provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of a process for obtaining disturbance data provided in an embodiment of the present application;

[0074] Figure 8A flowchart of a method for training a federated neural network according to another embodiment of the present application;

[0075] Figure 9 A schematic diagram of the structure of a federated neural network training device provided in an embodiment of the present application;

[0076] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0078] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0079] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0080] First, several terms involved in this application are introduced and explained:

[0081] Forward Activations, forward activation, that is, the intermediate result of forward propagation calculation.

[0082] Backward Derivatives, reverse derivative, that is, the gradient of the loss function with respect to the forward activation.

[0083] One of the key differences between federated learning and standard machine learning tasks is that the number of training participants increases from one to two or even multiple. Federated learning allows multiple parties to participate in the same model training task, completing the model training task without releasing data and protecting data privacy, thus breaking down data silos. Therefore, a key challenge lies in coordinating two or more parties to complete a model training task while protecting the data security of both parties and preventing either party from gaining access to the other's data. However, for the most common binary classification task in machine learning, existing methods for vertical federated neural network training either risk label information leakage through backward derivatives or suffer from inefficiencies due to the need to protect label information.

[0084] See Figure 1 , which exemplarily shows a flow chart of vertical federated neural network training in related technologies. As shown in the figure, in forward propagation, the participants run the following process:

[0085] Each participant uses its own feature data as input, performs forward propagation calculations on its own bottom model, and obtains forward activations, i.e. Z A and Z B ;

[0086] Participant A activates its own forward direction, i.e. Z A , sent to participant B;

[0087] Participant B will receive the forward activation Z A , and your own forward activation Z B , as input, perform forward propagation calculation of the top model (Top model) to obtain the model prediction value y';

[0088] For model training, participant B needs to use the predicted values and true labels to calculate the model loss based on the loss function (such as the Logistic loss function).

[0089] In order to iteratively train the model, it is necessary to perform backpropagation to calculate the model gradient and update the model. The specific operations are as follows:

[0090] Participant B calculates the back propagation calculation of the top model based on the model loss, calculates the gradient of the top model, and Z A and Z B The corresponding backward derivatives are and

[0091] Participant B will Sent to Party A;

[0092] Each participant calculates the backpropagation of its bottom model based on its own reverse partial derivative to obtain the gradient of the bottom model;

[0093] Each participant updates the model based on the model gradient (including the bottom and top model gradients).

[0094] Since the reverse partial derivative of participant A is It is calculated by back propagation. Studies have shown that for two-classification tasks, The label information is implicit in the To solve this problem, it is necessary to let participant B send Before, yes Add noise to perturb Then, Party B will Sent to Party A( will not be sent to Party A), Party A uses Perform back propagation calculations on the underlying model. However, how to perform perturbations is a key issue. A good perturbation algorithm should meet two conditions:

[0095] (1) Participant A cannot Invert the label information;

[0096] (2) The perturbation has little impact on the accuracy of the trained model.

[0097] The related art provides an iterative perturbation algorithm called Marvell. This work first defines a distance metric, namely the sum of KL divergences (sumKL), which approximately describes the distance that participant A can obtain from The probability of inferring label information in is defined as a mathematical optimization problem: given a perturbation budget, after perturbation to that extent, the minimum value that the sum of the KL divergences (sumKL) can reach is calculated.

[0098] The pseudo code of Marvell's algorithm can be expressed as:

[0099] Procedure (initial perturbation amount init_budget, maximum tolerable index max_sumKL):

[0100] |budget=int_budget

[0101] |Calculate the minimum sumKL that can be achieved under the current budget;

[0102] ||while sumKL>max_sumKL

[0103] ||Increase the minimum sumKL that can be achieved under the current budget;

[0104] |return the perturbation used to achieve the minimum sumKL under the current budget.

[0105] As shown in the pseudocode above, Marvell requires two parameters: the initial perturbation amount and the maximum acceptable distance indicator. The algorithm then iteratively increases the amount and returns the perturbation corresponding to the first time the condition is met.

[0106] There are two main problems with the existing technical solutions:

[0107] (i) Each time a budget is given, there is no closed-form solution for the corresponding minimum sumKL calculation, and each calculation of the minimum sumKL takes a long time;

[0108] (ii) The perturbation amount needs to be iteratively increased until the conditions are met. Given a maximum acceptable distance metric, it is impossible to directly calculate the corresponding perturbation. Due to these two issues, the algorithm takes a long time to calculate the perturbation, resulting in inefficient model training.

[0109] The federated neural network training method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0110] The inventive concept of the embodiments of the present application is as follows:

[0111] Since the reverse partial derivatives of positive and negative samples each show a certain regularity, we first model the multivariate Gaussian distribution of the reverse partial derivatives of positive and negative samples respectively. In order to perturb the reverse partial derivatives, we need to calculate the noise data distribution for the reverse partial derivatives of positive and negative samples. Based on the noise data distribution, the distribution of the reverse partial derivatives after adding the perturbation is difficult to distinguish. Therefore, the noise data distribution must meet two goals: 1) the covariance matrix of the distribution after the perturbation is an isotropic diagonal matrix, 2) the relative ratio of the expectation and the covariance matrix after the perturbation is to meet the degree of perturbation; based on the two goals, a covariance matrix that can calculate the noise data distribution at one time can be constructed, and the covariance matrix can be used to further obtain the value range of the noise data. Randomly obtain sampling noise from this value range, apply the sampling noise to the reverse partial derivative of the first participant, and obtain the reverse partial derivative with added noise. The first participant completes the backpropagation of the underlying model based on the reverse partial derivative with added noise.

[0112] The embodiment of the present application can add disturbances to the reverse partial derivative at one time, which greatly improves the efficiency of adding disturbances compared to the iterative method. When the customer gives a certain level of label protection (i.e., the level of disturbance), the disturbance that meets the protection requirements and has the least impact on model training can be quickly calculated, thereby improving the privacy and availability of vertical federated neural network model training.

[0113] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0114] The following describes an exemplary application of the training device for the federated neural network model of an embodiment of the present application. In one implementation, the training device for the federated neural network model provided by the embodiment of the present application can be implemented as any terminal such as a laptop computer, a tablet computer, a desktop computer, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), an intelligent robot, or an in-vehicle terminal. In another implementation, the training device for the federated neural network model provided by the embodiment of the present application can also be implemented as a server. The following describes an exemplary application of the training device for the federated neural network model when it is implemented as a server.

[0115] See Figure 2, which exemplarily shows an optional architectural diagram of a training system for a federated neural network model provided in an embodiment of the present application. The embodiment of the present application takes a two-party vertical federated neural network model as an example of a network model to be trained. To implement the training of the federated neural network model, the training system 10 for a federated neural network model provided in an embodiment of the present application includes a first terminal 100-1, a second terminal 100-2, a network 200, and a server 300, wherein the first terminal 100-1 is a terminal of the first participant, the second terminal 100-2 is a terminal of the second participant, and the first participant A (for example, an advertising company) and the second participant B (for example, a social networking platform) cooperate to complete the training of the federated neural network model, that is, jointly train one or more personalized recommendation models based on deep learning.

[0116] The first participant A has some data features, for example, (X1, X2, ..., X40), a total of 40-dimensional data features; and the second participant B may have data features or may not provide data features. For example, in one embodiment, the second participant B has (X41, X42, ..., X100), a total of 60-dimensional data features. The first participant A and the second participant B together have 100-dimensional data features, so the feature dimension of the training data is significantly expanded. For supervised deep learning, the first participant A and / or the second participant B also have label information Y of the training data. In this embodiment of the application, only the second participant B is used for label information (in binary classification, samples with a label of 1 are called positive samples, and samples with a label other than 1 are called negative samples).

[0117] Before model training, the first participant A sends the first part of sample data to the server 300 through the network 200 via the first terminal 100-1, and the second participant B sends the second part of sample data to the server 300 through the network 200 via the second terminal 100-2. After obtaining the sample data, the server starts model training and inputs the sample data into the federated neural network model. Each participant uses its own feature data as input and enters its own bottom-level model for forward activation. The first participant A sends its own forward activation to the interaction layer, and uses the forward activation of each participant as the input of the top-level model of the second participant to perform forward propagation calculation of the top-level model, and obtains The model predicts the value, uses the predicted value and the true label, and calculates the model loss according to the loss function. The second participant B calculates the back propagation calculation of the top model based on the model loss, calculates the gradient of the top model, and the reverse partial derivative corresponding to each participant, randomly samples noise data from a predetermined distribution range, adds noise data to the reverse partial derivative, obtains the perturbed reverse partial derivative, inputs the perturbed reverse partial derivative into the underlying model of the first participant, and performs back propagation calculation on the underlying model of the first participant to obtain the gradient of the underlying model. According to the model gradient (including the model gradients of the bottom and top layers), the model is updated to complete the training of the federated neural network model and obtain the trained federated neural network model.

[0118] The training method of the federated neural network model provided in the embodiments of the present application also relates to the field of artificial intelligence technology, and the training of the federated neural network model is achieved through artificial intelligence technology, or information recommendation based on the trained federated neural network model is achieved through artificial intelligence technology. In the embodiments of the present application, at least machine learning and natural language processing technologies in artificial intelligence technology can be used to achieve this. Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by formula. In the embodiments of the present application, machine learning technology is used to respond to network structure search requests to automatically search for the target network structure, and to train and optimize the controller and score model. Natural language processing (NLP) is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies

[0119] The training method of the federated neural network model provided in the embodiment of the present application can also be based on a cloud platform and implemented through cloud technology. For example, the above-mentioned server 300 can be a cloud server.

[0120] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology, all of which are applied in the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as those for video sites, image sites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0121] The embodiments of the present application can also be implemented using cloud storage technology within cloud technology. For example, when training a federated neural network model, the updated interaction layer model parameters can be stored in cloud storage, or the trained federated neural network model can be stored in cloud storage. In this way, when the federated neural network model is subsequently used, the model can be directly retrieved from the cloud storage for use in data processing, eliminating the need to retrain the federated neural network model, thereby improving data processing efficiency.

[0122] It should be noted here that cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0123] In an embodiment of the present application, a method for training a federated neural network is provided. The federated neural network model includes at least a bottom model of a first participant, an interaction layer, and a top model of a second participant, such as Figure 3 As shown, the training method includes:

[0124] S101. Perform reverse partial derivative of the batch of first-party data output by the top-level model.

[0125] It can be understood that steps S101 and S102 occur in the back propagation process. It can be seen from the above embodiment that the embodiment of the present application also includes a forward propagation process before step S101, that is, the first participant (for the sake of ease of description, the second participant in the embodiment of the present application does not have feature data, so the second participant does not have a corresponding underlying model) inputs its own feature data (that is, the first participant's data) into the underlying model for forward propagation calculation to obtain forward activation.

[0126] The number of first participants in the embodiment of the present application is an integer not less than 1, and each first participant has a corresponding underlying model.

[0127] The forward activation of the first participant is input to the interaction layer, and then input to the top model of the second participant for forward propagation calculation to obtain the model prediction value. The model loss is further calculated based on the model prediction value and the label set by the second participant through the loss function of the second participant.

[0128] In the back-propagation process, the top-level model is first used to obtain the gradient of the top-level model and the reverse partial derivative of the first-party data according to the model loss.

[0129] S102. Determine the positive samples and negative samples in the first participant's data based on the label information set by the second participant for the first participant's data, and determine the distribution range of the noise data based on the distribution of the reverse partial derivatives of the positive samples and the negative samples and a preset disturbance degree.

[0130] As can be seen from the above embodiment, the second participant has a label corresponding to the first participant's data. Therefore, each first participant's data can be determined as a positive sample or a negative sample based on the label. For example, the first participant's data with a label of 1 can be used as a positive sample, and the first participant's data with a label of 0 can be used as a negative sample. The distribution range of the embodiment of the present application is obtained based on the positive and negative sample distribution ranges and the preset disturbance degree. The distribution range can be specifically characterized by the Gaussian distribution fitted by the reverse partial derivative. The disturbance degree of the embodiment of the present application can be set by the user, that is, the second participant. The greater the disturbance degree, the less obvious the difference in reverse partial derivatives between the positive and negative samples after disturbance. The disturbance degree of the embodiment of the present application is similar to the distance indicator sumKL in the Marvell algorithm.

[0131] Since positive and negative samples will show a certain regularity, the embodiment of the present application integrates the distribution information of positive and negative samples, and combines the disturbance degree to obtain the distribution range of the noise data. For any sample data, it is only necessary to randomly sample from the distribution range to obtain noise data that meets the given disturbance degree. Compared with the iterative method of obtaining noise data, the efficiency is greatly improved.

[0132] S103. Randomly sample noise data from the distribution range for each reverse partial derivative, add the noise data to the reverse partial derivative, and obtain a perturbed reverse partial derivative.

[0133] The embodiment of the present application randomly samples noise data from a predetermined distribution range for each reverse derivative in a batch of reverse derivatives. That is, the disturbance added to each reverse derivative is random, rather than adding the same disturbance to multiple reverse derivatives in the same batch. This processing can further increase the difficulty for the first participant to infer the label information.

[0134] S104: Input the perturbed reverse partial derivative output by the interaction layer into the underlying model, so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative.

[0135] The perturbed reverse partial derivative is input into the underlying model of the first participant, and the underlying model is back-propagated to obtain the gradient of the underlying model, and the parameters of the underlying model and the top model are further updated according to the gradient.

[0136] The training method of the federated neural network of the embodiment of the present application, wherein the federated neural network model includes a bottom model of the first participant and a top model of the second participant, and during the back propagation process, the reverse partial derivative of the first participant's data output by the top model is determined, and according to the label information set by the second participant for the first participant's data, after determining a small batch of positive and negative samples, the distribution range of the noise data can be quickly determined according to the distribution of the reverse partial derivatives of the positive and negative samples and the preset disturbance degree. Each time the disturbance is added to the reverse partial derivative, the noise data can be randomly sampled from the distribution range to be scrambled, without the need for iteration or long-term complex calculations like the existing technology, and is more efficient than related technologies. Furthermore, when the efficiency is improved, a lower maximum acceptable distance index can be provided, thereby improving the protection capability of the label.

[0137] Based on the above embodiments, as an optional embodiment, see Figure 4 , which exemplarily shows a flow chart of determining the distribution range of noise data in an embodiment of the present application, including:

[0138] S201, determining a difference between a first expectation and a second expectation;

[0139] By modeling the reverse partial derivatives of multiple positive samples and negative samples sampled in a small batch respectively, we can obtain the first multivariate Gaussian distribution of the reverse partial derivatives of the positive samples and the second multivariate Gaussian distribution of the reverse partial derivatives of the negative samples.

[0140] Figure 5This is a schematic diagram of the multivariate Gaussian distribution for modeling the reverse partial derivatives of positive and negative samples in an embodiment of the present application. As shown in the figure, the hidden dimension of the underlying model in the figure is 2, so the coordinate system in the figure is a two-dimensional coordinate system, where the reverse partial derivatives of the positive samples are distributed in the second quadrant, while the reverse partial derivatives of the negative samples are distributed in the first and fourth quadrants. The distribution of the reverse partial derivatives of the positive and negative samples has obvious differences. By constructing the corresponding multivariate Gaussian distributions respectively, it can be found that the two multivariate Gaussian distributions have only a small overlap.

[0141] The first multivariate Gaussian distribution P1 of the embodiment of the present application can be expressed as: P1~N(μ1,Σ1), and the second multivariate Gaussian distribution P0 of the reverse partial derivative of the negative sample is expressed as: P1~N(μ1,Σ1), where the second μ0 and the first expected μ1 are vectors of size d. If a complete covariance matrix is used, subsequent calculations will be more complicated. Therefore, the first covariance matrix Σ1 and the second covariance matrix Σ0 in the embodiment of the present application are diagonal matrices of size d×d, where d represents the hidden dimension of the reverse partial derivative.

[0142] In actual operation, μ0, μ1, ∑1, ∑0 can be estimated based on the reverse partial derivatives of each mini-batch. For example, μ1 is obtained by calculating the average value of the reverse partial derivatives of all positive samples, and ∑1 is obtained by calculating the variance of the reverse partial derivatives of all positive samples. Similarly, μ0 is obtained by calculating the average value of the reverse partial derivatives of all negative samples, and ∑0 is obtained by calculating the variance of the reverse partial derivatives of all negative samples.

[0143] It is worth noting that the Marvell algorithm in the related art not only assumes that the covariance matrix is a diagonal matrix, but also makes an isotropic assumption, that is, the elements on the diagonal are the same (∑1=δ1I d×d ,∑0=δ0I d×d , where δ1, δ0 are constants, I d×d is the unit matrix). However, the algorithm proposed in the embodiment of the present application does not require the isotropy assumption and is more in line with the actual situation.

[0144] S202: Obtain the covariance matrix of the first participant's data after reverse partial derivative perturbation according to the difference and a preset disturbance degree.

[0145] In order to perturb the reverse partial derivative, the embodiment of the present application needs to calculate a distribution range of noise data for the reverse partial derivatives of positive samples and negative samples respectively, that is, See Figure 6, which exemplarily shows a schematic diagram of the multivariate Gaussian distribution before and after adding perturbations to the reverse partial derivatives of positive and negative samples in the embodiment of the present application. It can be seen from the figure that after adding perturbations, the difference between the multivariate Gaussian distributions of positive and negative samples (y1 and y0 represent the multivariate Gaussian distribution of positive samples and the multivariate Gaussian distribution of negative samples, respectively) is obviously no longer significant, and there is a large overlap between the two multivariate Gaussian distributions. The embodiment of the present application hopes that the two distributions after adding perturbations will be Difficult to distinguish, thus achieving the goal of protecting the label.

[0146] In order to make the distribution of the reverse partial derivatives of the perturbed positive and negative samples difficult to distinguish, the embodiment of the present application requires that the distribution of the noise data meet the following two goals:

[0147] 1) The covariance matrix of the perturbed multivariate Gaussian distribution of the positive samples should be equal to the covariance matrix of the perturbed multivariate Gaussian distribution of the negative samples, and also equal to the product of the preset coefficients (i.e., the values on the diagonal of the covariance matrix of the perturbed multivariate Gaussian distribution) and the identity matrix. The first goal is to make the covariance matrix of the perturbed distribution an isotropic diagonal matrix, making it indistinguishable.

[0148] 2) The ratio of the difference between the first expectation and the second expectation to the covariance matrix of the perturbed distribution should be measured using a predetermined degree of perturbation.

[0149] Therefore, based on the second objective, the embodiment of the present application can obtain the covariance matrix after disturbance through the difference and the predetermined disturbance relationship.

[0150] S203 : Obtain a covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix, and the second covariance matrix.

[0151] After obtaining the covariance matrix after disturbance, based on the first goal, the covariance matrix of the noise data distribution can be obtained by calculating the difference between the covariance matrix of the cave post-wash and the first covariance matrix and the second covariance matrix.

[0152] S204 , modeling is performed according to the covariance matrix of the noise data distribution and the preset third expectation to obtain the distribution range of the noise data.

[0153] It should be understood that two parameters are required to describe the Gaussian distribution: expectation and variance. The first expectation and the second expectation of the embodiment of the present application are the expectations of the reverse partial derivatives of the positive samples and the negative samples, respectively. The third expectation is the expectation of the distribution range of the noise data. The embodiment of the present application does not impose any specific restrictions on the size of the third expectation.

[0154] Based on the above embodiments, as an optional embodiment, the difference between the first expectation and the second expectation includes sub-differences between the first expectation and the second expectation in each hidden layer dimension of the reverse partial derivative.

[0155] According to the difference and the predetermined disturbance degree, the covariance matrix after disturbance is obtained, including:

[0156] Each sub-difference in the difference is squared and summed up. The covariance matrix after perturbation is obtained based on the summation result, the identity matrix and the degree of perturbation.

[0157] The calculation formula for the sum of the squares of each sub-difference in all differences in this application can be expressed as: Among them, m i represents the sub-difference between the first expectation and the second expectation in the i-th hidden layer dimension of the backward partial derivative.

[0158] The second goal that the noise data distribution needs to meet is the relative ratio of the expectation of the reverse partial derivative after perturbation to the covariance matrix, which satisfies the predetermined degree of perturbation.

[0159]

[0160] Further, according to the summed result, the unit matrix and the disturbance degree, the covariance matrix after the disturbance is obtained, including: calculating the quotient between the summed result and the disturbance degree, and multiplying the quotient by the unit matrix as the covariance matrix after the disturbance

[0161] When setting the maximum acceptable disturbance level D * Then, by substituting the above formula, we can obtain the diagonal value of the covariance matrix after disturbance. Multiplying it with the identity matrix gives the perturbed covariance matrix:

[0162] Based on the above embodiments, as an optional embodiment, obtaining the covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix of the first multivariate Gaussian distribution, and the second covariance matrix of the second multivariate Gaussian distribution includes:

[0163] From the first goal that the noise data distribution needs to meet, we can know that the covariance matrix of the noise data distribution is satisfy That is, the covariance matrix of the perturbed distribution is an isotropic diagonal matrix, making it indistinguishable. Therefore, after obtaining the perturbed covariance matrix, the covariance matrix of the noise data for the positive sample can be obtained by taking the difference between the perturbed covariance matrix and the first covariance matrix and the second covariance matrix. (third covariance matrix) and the covariance matrix of the noise data for negative samples (the fourth covariance matrix), please refer to the following formula for details:

[0164] in

[0165] Based on the above embodiments, as an optional embodiment, modeling is performed according to the covariance matrix of the noise data distribution and the preset expectation to obtain the distribution range of the noise data, including:

[0166] According to the third covariance matrix and the preset expectation modeling, a first distribution range of the noise data is obtained; according to the third covariance matrix and the preset expectation modeling, a second distribution range of the noise data is obtained.

[0167] Specifically, the third expectation in the embodiment of the present application may be 0, that is, the distribution range of the noise data includes: and Wherein, η1 represents the first distribution range of the noise data for the positive sample, and η0 represents the second distribution range of the noise data for the negative sample.

[0168] Based on the above embodiments, as an optional embodiment, adding noise data to the reverse partial derivative to obtain the perturbed reverse partial derivative includes:

[0169] S301, randomly sampling first noise data from a first distribution range, and randomly sampling second noise data from a second distribution range;

[0170] S302, performing weighted summation on the first noise data and the second noise data to obtain disturbance data;

[0171] S303: Add perturbation data to the reverse partial derivative to obtain the perturbed reverse partial derivative.

[0172] Specifically, the reverse partial derivative after perturbation can be expressed by the following formula:

[0173]

[0174] in, represents the reverse partial derivative of the jth first participant data; α1 and α0 represent the first noise and the second noise As can be seen from the formula, the embodiment of the present application will separately sample noise data for the reverse partial derivative of each first-party data, and add disturbance based on the noise data, rather than sampling the same noise and using it on multiple samples.

[0175] Based on the above embodiments, as an optional embodiment, if the first participant data corresponding to the reverse partial derivative is a positive sample, the weight corresponding to the first noise data is set to a first value, and the weight corresponding to the second noise data is set to a second value;

[0176] If the first participant data corresponding to the reverse partial derivative is a negative sample, the weight corresponding to the first noise data is set to the second value, and the weight corresponding to the second noise data is set to the first value;

[0177] The first value is greater than the second value.

[0178] For example, the first value may be 1 and the second value may be 0. The embodiment of the present application does not further limit the specific sizes of the first value and the second value.

[0179] See Figure 7 , which exemplarily shows a schematic diagram of a process for obtaining disturbance data according to an embodiment of the present application, as shown in the figure, including:

[0180] Determining positive samples and negative samples in the first party data based on labels of the batch of first party data;

[0181] Fit the reverse partial derivative of the positive sample and the reverse partial derivative of the negative sample respectively to obtain the first multivariate Gaussian distribution and the second multivariate Gaussian distribution;

[0182] determining a sub-difference between a first expectation of a first multivariate Gaussian distribution and a second expectation of a second multivariate Gaussian distribution, along each hidden dimension of the underlying model;

[0183] Each sub-difference in the difference is squared and summed up. The covariance matrix after perturbation is obtained based on the summed result, the identity matrix, and the degree of perturbation.

[0184] Calculating the differences between the covariance matrix after disturbance and the first covariance matrix and the second covariance matrix respectively to obtain a third covariance matrix for the noise data distribution of the positive sample and a fourth covariance matrix for the noise data distribution of the negative sample;

[0185] Modeling based on the third covariance matrix and the preset expectation to obtain a first distribution range of the noise data for the positive sample; modeling based on the fourth covariance matrix and the preset expectation to obtain a second distribution range of the noise data for the negative sample;

[0186] Randomly sample first noise data from a first distribution range, and randomly sample second noise data from a second distribution range;

[0187] Determine, based on whether the first participant data corresponding to the reverse partial derivative is a positive sample or a negative sample, a first weight and a second weight corresponding to the first noise data and the second noise data, respectively;

[0188] The first noise data and the second noise data are weightedly summed according to the first weight and the second weight to obtain disturbance data.

[0189] See Figure 8 , which exemplarily shows a flow chart of a training method for a federated neural network according to another embodiment of the present application. The federated neural network according to the embodiment of the present application further includes an interaction layer, which scrambles the reverse derivative of the first participant's data, making the scrambling process more independent and convenient for other first participants to call. As shown in the figure, it includes:

[0190] S401, the first participant A uses his own feature data as input and performs forward propagation calculation of his own underlying model to obtain the forward activation Z A ;

[0191] S402, the first participant A activates his forward Z A , sent to the interaction layer;

[0192] S403, the interaction layer will receive the forward activation Z A As input, it is sent to the forward propagation calculation of the top model of the second participant B to obtain the model prediction value y';

[0193] S404, the second participant B uses the model prediction value y' and the true label to calculate the model loss according to the loss function, calculates the back propagation calculation of the top model according to the model loss, calculates the gradient of the top model, and Z A The corresponding reverse partial derivative

[0194] S405: The second participant B inputs the reverse partial derivative of the batch of the first participant's data into the interaction layer. The interaction layer determines the positive and negative samples in the batch of the first participant's data based on the labels set by the second participant B.

[0195] S406. Fit the reverse partial derivatives of the positive sample and the negative sample respectively to obtain a first multivariate Gaussian distribution: P1~N(μ1,∑1) and a second multivariate Gaussian distribution: P1~N(μ1,∑1);

[0196] S407: Calculate the difference m between the first expectation and the second expectation in each hidden dimension, and square the difference in each hidden dimension and sum it to obtain

[0197] S408: Determine the disturbance level set by the second participant B, that is, the maximum acceptable disturbance level D * , combined with Get the diagonal values of the perturbed covariance matrix

[0198] S409, will and the identity matrix I d×d Multiply them together to obtain the covariance matrix after disturbance;

[0199] S410, according to the formula: Get the covariance matrix of the noise data for the positive sample (third covariance matrix) and the covariance matrix of the noise data for negative samples (fourth covariance matrix);

[0200] S411. Obtain two distribution ranges of noise data: and Wherein, η1 represents the first distribution range of the noise data for the positive sample, and η0 represents the second distribution range of the noise data for the negative sample;

[0201] S412: For the reverse partial derivative of the i-th first participant data, randomly sample from η1 and η0 to obtain noise data and Weight the two noise data separately and add the weighted results to the reverse partial derivative to obtain the perturbed reverse partial derivative. Right now

[0202] S413, reverse partial derivative after disturbance The underlying model sent to the first participant A is back-propagated to obtain the gradient of the underlying model, and the model parameters are updated according to the model gradient (including the model gradients of the bottom and top layers).

[0203] The present application embodiment provides a training device for a federated neural network, such as Figure 9 As shown, the federated neural network model includes at least the bottom model of the first participant and the top model of the second participant. The training device may include: a data acquisition module 901, a noise range determination module 902, a noise addition module 903 and a parameter update module 904, wherein,

[0204] The data acquisition module 901 is used to determine the reverse partial derivative input of the batch of first-party data output by the top-level model into the interaction layer;

[0205] Noise range determination module 902, configured to determine positive and negative samples in the first party's data based on the label information set by the second party for the first party's data, and determine the distribution range of the noise data based on the distribution of the reverse partial derivatives of the positive and negative samples and a preset disturbance degree;

[0206] A noise adding module 903 randomly samples noise data from a distribution range for each reverse partial derivative, adds the noise data to the reverse partial derivative, and obtains a perturbed reverse partial derivative;

[0207] The parameter updating module 904 is used to input the perturbed reverse partial derivative output by the interaction layer into the underlying model, so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative.

[0208] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0209] Compared with related technologies, it can achieve:

[0210] By determining the reverse partial derivative of the first participant's data output by the top-level model, and based on the label information set by the second participant for the first participant's data, after determining a small batch of positive and negative samples, the distribution range of the noise data can be quickly determined based on the distribution of the reverse partial derivatives of the positive and negative samples and the preset degree of disturbance. Each time a disturbance is added to the reverse partial derivative, noise data can be randomly sampled from the distribution range for scrambling. This does not require iteration or long-term complex calculations like in existing technologies, and is more efficient than related technologies. Furthermore, when the efficiency is improved, a lower maximum acceptable distance indicator can be provided, thereby improving the protection capability of the label.

[0211] As an optional embodiment, the noise range determination module includes:

[0212] a difference calculation submodule, configured to determine a difference between the first expectation and the second expectation;

[0213] A post-perturbation covariance submodule is used to obtain the covariance matrix of the batch of first-party data after reverse partial derivative perturbation based on the difference and a predetermined degree of perturbation;

[0214] A noise covariance submodule, configured to obtain a covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix, and the second covariance matrix;

[0215] The disturbance distribution submodule is used to perform modeling based on the covariance matrix of the noise data distribution and the preset third expectation to obtain the distribution range of the noise data;

[0216] Among them, the first expectation and the first covariance matrix are the expectation and covariance matrices of the first multivariate Gaussian distribution modeled according to the reverse partial derivative of the positive sample, and the second expectation and the second covariance matrix are the expectation and covariance matrices of the second multivariate Gaussian distribution modeled according to the reverse partial derivative of the negative sample.

[0217] As an optional embodiment, the difference includes sub-differences between the first expectation and the second expectation in each hidden dimension of the underlying model;

[0218] The perturbed covariance submodule is specifically used to square each sub-difference in the difference and then sum them up, and obtain the perturbed covariance matrix based on the summed result, the unit matrix and the perturbation degree.

[0219] As an optional embodiment, the post-disturbance covariance submodule is specifically used to calculate the quotient between the summed result and the disturbance degree, and use the product of the quotient and the identity matrix as the post-disturbance covariance matrix.

[0220] As an optional embodiment, the noise covariance submodule is specifically used to: calculate the difference between the perturbed covariance matrix and the first covariance matrix and the second covariance matrix, respectively, to obtain a third covariance matrix for the noise data distribution of positive samples and a fourth covariance matrix for the noise data distribution of negative samples.

[0221] As an optional embodiment, the disturbance distribution submodule includes:

[0222] A positive sample noise distribution unit, configured to obtain a first distribution range of noise data for the positive sample based on a third covariance matrix and a preset expectation model;

[0223] The negative sample noise distribution unit is used to obtain a second distribution range of noise data for negative samples according to the fourth covariance matrix and preset expectation modeling.

[0224] As an optional embodiment, the noise adding module includes:

[0225] A noise data sampling submodule, configured to randomly sample first noise data from a first distribution range and randomly sample second noise data from a second distribution range;

[0226] A weighting submodule, configured to perform weighted summation on the first noise data and the second noise data to obtain disturbance data;

[0227] The disturbance adding submodule is used to add disturbance data to the reverse partial derivative to obtain the perturbed reverse partial derivative.

[0228] As an optional embodiment, the weighting submodule is specifically configured to:

[0229] If the first participant data corresponding to the reverse partial derivative is a positive sample, the weight corresponding to the first noise data is set to the first value, and the weight corresponding to the second noise data is set to the second value;

[0230] If the first participant data corresponding to the reverse partial derivative is a negative sample, the weight corresponding to the first noise data is set to the second value, and the weight corresponding to the second noise data is set to the first value;

[0231] The first value is greater than the second value.

[0232] As an optional embodiment, the first expectation and the second expectation are both vectors of dimension d; the first covariance matrix and the second covariance matrix are both d×d diagonal matrices; where d is the hidden layer dimension of the underlying model.

[0233] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the federated neural network training method. Compared with the related art, the following can be achieved:

[0234] By determining the reverse partial derivative of the first participant's data output by the top-level model, and based on the label information set by the second participant for the first participant's data, after determining a small batch of positive and negative samples, the distribution range of the noise data can be quickly determined based on the distribution of the reverse partial derivatives of the positive and negative samples and the preset degree of disturbance. Each time a disturbance is added to the reverse partial derivative, noise data can be randomly sampled from the distribution range for scrambling. This does not require iteration or long-term complex calculations like in existing technologies, and is more efficient than related technologies. Furthermore, when the efficiency is improved, a lower maximum acceptable distance indicator can be provided, thereby improving the protection capability of the label.

[0235] In an alternative embodiment, an electronic device is provided, such as Figure 10 As shown, Figure 10 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0236] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0237] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0238] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0239] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0240] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented. Compared with the related art, the following can be implemented:

[0241] By determining the reverse partial derivative of the first participant's data output by the top-level model, and based on the label information set by the second participant for the first participant's data, after determining a small batch of positive and negative samples, the distribution range of the noise data can be quickly determined based on the distribution of the reverse partial derivatives of the positive and negative samples and the preset degree of disturbance. Each time a disturbance is added to the reverse partial derivative, noise data can be randomly sampled from the distribution range for scrambling. This does not require iteration or long-term complex calculations like in existing technologies, and is more efficient than related technologies. Furthermore, when the efficiency is improved, a lower maximum acceptable distance indicator can be provided, thereby improving the protection capability of the label.

[0242] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor. Compared with the related art, it can be achieved: by determining the reverse partial derivative of the first participant data output by the top-level model, according to the label information set by the second participant for the first participant data, after determining a small batch of positive and negative samples, the distribution range of the noise data can be quickly determined according to the distribution of the reverse partial derivatives of the positive samples and negative samples and the preset disturbance degree. Each time the disturbance is added to the reverse partial derivative, the noise data can be randomly sampled from the distribution range for scrambling. There is no need for iteration or long-term complex calculations like the existing technology, which is more efficient than the related art. Furthermore, when the efficiency is improved, a lower maximum acceptable distance index can be provided, thereby improving the protection capability of the label.

[0243] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0244] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0245] The above are only optional implementation methods for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A method for training a federated neural network, characterized in that: The federated neural network model includes at least a bottom model of a first participant and a top model of a second participant, and the training method includes: During backpropagation, determining the reverse partial derivative of the batch of first-party data output by the top model; Determining, based on the label information set by the second party for the data of the first party, positive samples and negative samples in the data of the first party, and determining, based on the distribution of the reverse partial derivatives of the positive samples and the negative samples and a preset disturbance degree, a distribution range of the noise data; Randomly sampling noise data from the distribution range for each reverse partial derivative, adding the noise data to the reverse partial derivative, and obtaining a perturbed reverse partial derivative; The perturbed reverse partial derivative is input into the underlying model, so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative.

2. The training method according to claim 1, characterized in that The determining the distribution range of the noise data according to the distribution of the reverse partial derivatives of the positive samples and the negative samples and a preset disturbance degree includes: Determine the difference between the first expectation and the second expectation; Obtaining a covariance matrix of the first participant's data after reverse partial derivative perturbation based on the difference and a preset disturbance degree; Obtaining a covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix, and the second covariance matrix; Modeling is performed according to the covariance matrix of the noise data distribution and a preset third expectation to obtain the distribution range of the noise data; Among them, the first expectation and the first covariance matrix are respectively the expectation and covariance matrices of the first multivariate Gaussian distribution modeled according to the reverse partial derivative of the positive sample, and the second expectation and second covariance matrices are respectively the expectation and covariance matrices of the second multivariate Gaussian distribution modeled according to the reverse partial derivative of the negative sample.

3. The training method according to claim 2, characterized in that The difference includes sub-differences between the first expectation and the second expectation in each hidden dimension of the underlying model; The step of obtaining a covariance matrix after disturbance according to the difference and a predetermined disturbance degree includes: Each sub-difference in the difference is squared and then summed, and the covariance matrix after the disturbance is obtained according to the summed result, the identity matrix and the disturbance degree.

4. The training method according to claim 3, characterized in that The step of obtaining the covariance matrix after the disturbance according to the summation result, the identity matrix, and the disturbance degree includes: A quotient of the summed result and the disturbance degree is calculated, and the product of the quotient and the identity matrix is used as the covariance matrix after the disturbance.

5. The training method according to claim 2, characterized in that: The step of obtaining the covariance matrix of the noise data distribution according to the disturbed covariance matrix, the first covariance matrix of the first multivariate Gaussian distribution, and the second covariance matrix of the second multivariate Gaussian distribution comprises: The differences between the covariance matrix after the disturbance and the first and second covariance matrices are calculated respectively to obtain a third covariance matrix for the noise data distribution of the positive samples and a fourth covariance matrix for the noise data distribution of the negative samples.

6. The training method according to claim 5, characterized in that Modeling is performed according to the covariance matrix of the noise data distribution and a preset expectation to obtain the distribution range of the noise data, including: Obtaining a first distribution range of noise data for positive samples based on the third covariance matrix and preset expectation modeling; According to the fourth covariance matrix and preset expected modeling, a second distribution range of the noise data for the negative sample is obtained.

7. The training method according to claim 6, characterized in that Adding noise data to the reverse partial derivative to obtain a disturbed reverse partial derivative includes: Randomly sampling first noise data from the first distribution range, and randomly sampling second noise data from the second distribution range; Performing weighted summation on the first noise data and the second noise data to obtain disturbance data; The perturbation data is added to the reverse partial derivative to obtain the perturbed reverse partial derivative.

8. The training method according to claim 7, characterized in that: The weighted summing of the first noise data and the second noise data includes: If the first participant data corresponding to the reverse partial derivative is a positive sample, the weight corresponding to the first noise data is set to a first value, and the weight corresponding to the second noise data is set to a second value; If the first participant data corresponding to the reverse partial derivative is a negative sample, the weight corresponding to the first noise data is set to the second value, and the weight corresponding to the second noise data is set to the first value; The first value is greater than the second value.

9. The training method according to claim 2, characterized in that: The first expectation and the second expectation are both vectors of dimension d; the first covariance matrix and the second covariance matrix are both d×d diagonal matrices; where d is the hidden layer dimension of the underlying model.

10. A training device for a federated neural network, characterized in that: The federated neural network model includes at least a bottom model of a first participant and a top model of a second participant, and the training device includes: A data acquisition module, configured to determine the reverse partial derivative of the batch of first-party data output by the top-level model during the back-propagation process; a noise range determination module, configured to determine positive samples and negative samples in the first party's data based on label information set by the second party for the first party's data, and determine a distribution range of the noise data based on the distribution of reverse partial derivatives of the positive and negative samples and a preset disturbance degree; a noise adding module, configured to randomly sample noise data from the distribution range for each reverse partial derivative, add the noise data to the reverse partial derivative, and obtain a disturbed reverse partial derivative; A parameter updating module is used to input the perturbed reverse partial derivative into the underlying model so that the underlying model updates the parameters of the underlying model according to the perturbed reverse partial derivative.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the training method of the federated neural network according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for training a federated neural network according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the training method of the federated neural network according to any one of claims 1 to 9 are implemented.

Images