Method and apparatus for calculating activation function in neural network, and computing device

By adopting a combination of repetitive secret sharing and function secret sharing in neural networks, the calculation amount and communication volume of nonlinear activation functions are reduced, and the problem of low computational efficiency of activation functions in the existing technology is solved, and efficient neural Network training and secure collaborative training of multi-party data.

WO2024234682A1PCT designated stage expired Publication Date: 2024-11-21HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/070871
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-07-26
Filing Date
2024-01-05
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

In the prior art, in neural network training, the calculation efficiency of the nonlinear activation function is low, resulting in excessive communication volume and calculation volume, affecting the model training efficiency.

Method used

Using a combination of repetitive secret sharing and function secret sharing, the calculation amount and communication volume of nonlinear activation functions in neural networks is reduced, and the activation function is realized by decomposing the activation function and sharing the ciphertext and function share among the calculation participants. Efficient calculation.

Benefits of technology

It effectively reduces the traffic and computing volume during neural network training, improves the efficiency of model training, and ensures the security of multi-party data privacy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024070871_21112024_PF_FP_ABST
    Figure CN2024070871_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A method and apparatus for calculating an activation function in a neural network, and a computing device. The method comprises: a first calculation participant, a second calculation participant and a third calculation participant respectively obtaining shares of a first input; the first calculation participant and the second calculation participant respectively performing calculation according to their respective obtained shares of the first input, so as to obtain ciphertexts of the first input; the first calculation participant determining a first function result according to the ciphertext of the first input and a first sub-function; the second calculation participant determining a second function result according to the ciphertext of the first input and a second sub-function; and the first calculation participant, the second calculation participant and the third calculation participant respectively performing calculation according to their respective obtained shares of the first function result and respective obtained shares of the second function result, so as to obtain shares of a first activation function result. The method can reduce the communication traffic and the amount of calculation which are required during neural network training.
Need to check novelty before this filing date? Find Prior Art

Description

Method, device and computing equipment for calculating activation function in neural network

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 16, 2023, with application number 202310551642.X and invention name “Method for multi-party secure and efficient neural network training supported by semi-trusted cloud”, and the Chinese patent application filed on July 26, 2023, with application number 202310926879.1 and invention name “Method, device and computing device for calculating activation function in neural network”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of secure computing, and more specifically, to a method, apparatus, and computing device for calculating activation functions in a neural network. Background Art

[0003] Secure multi-party learning is a distributed machine learning paradigm that allows multiple parties to collaboratively train AI models using their own data, without aggregating data from multiple parties (organizations or users). Traditional machine learning paradigms require the aggregation of large amounts of data for model training, and this training data is likely to come from multiple different organizations or users. Aggregating data from multiple different organizations or users poses a significant risk of data leakage, exposing information assets for organizations and potentially compromising personal privacy for individual users.

[0004] These issues pose significant challenges to AI model training. To address these challenges, secure multi-party learning (SML) technology has emerged. While ensuring data privacy, SML leverages secure multi-party computing techniques such as secret sharing and obfuscated circuits to collaboratively train AI models. This technology protects user data from multiple parties while leveraging their data to collaboratively train models, resulting in more robust models.

[0005] In a related technical solution, a secure multi-party learning framework includes three computing participants. These three computing participants perform matrix multiplication operations in the linear phase through repeated secret sharing. When performing the activation function calculation in the nonlinear phase, the nonlinear activation function is converted into a linear calculation and executed through repeated secret sharing. Due to the large amount of computation required for linear calculation, the overall efficiency is low.

[0006] Therefore, how to reduce the amount of communication and computing required for neural network training has become a technical problem that needs to be solved urgently.

[0007] Summary of the Invention

[0008] The present application provides a method, apparatus, and computing device for calculating activation functions in a neural network, which can reduce the communication and computing workload required for neural network training.

[0009] In a first aspect, a method for calculating an activation function in a neural network is provided, the method comprising: a first calculation participant, a second calculation participant, and a third calculation participant respectively obtain a share of a first input, wherein the first input is input information of a first activation function, and the share of the first input is to share the first input among the first calculation participant, the second calculation participant, and the third calculation participant in a secret sharing manner; the first calculation participant and the second calculation participant respectively calculate a ciphertext of the first input based on their respective obtained shares of the first input; the first calculation participant determines a first function result based on the ciphertext of the first input and a first sub-function, wherein the first sub-function is a function share obtained by splitting the first activation function in a function secret sharing manner, and the first function The result is the output of the first sub-function; the second computing participant determines the second function result based on the ciphertext of the first input and the second sub-function, wherein the second sub-function is another function share obtained by splitting the first activation function in a manner of secret sharing of the function, and the second function result is the output of the second sub-function; the first computing participant, the second computing participant and the third computing participant respectively calculate the share of the first activation function result based on their respective shares of the first function result and their respective shares of the second function result, and the combination of the shares of the first activation function results calculated by the first computing participant, the second computing participant and the third computing participant can restore the first activation function result, which is the output of the first activation function.

[0010] In the above technical solution, for the calculation of the activation function in the nonlinear stage, the communication volume and computational complexity required for neural network training can be reduced by combining secret sharing and function secret sharing.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the share of the first input is to share the first input among the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner.

[0012] In conjunction with the first aspect, in certain implementations of the first aspect, the method further includes: the third computing participant, in an offline phase, splitting the first activation function using the function secret sharing method to obtain the first sub-function and the second sub-function. The third computing participant sends the first sub-function and the second sub-function to the first computing participant and the second computing participant, respectively.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the first computing participant shares the first function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the first function result; the second computing participant shares the second function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the second function result.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the first computing participant shares the first function result with the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner, and the second computing participant shares the second function result with the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the first computing participant obtains a first share of the first activation function result based on its share of the first function result and its share of the second function result; the second computing participant obtains a second share of the first activation function result based on its share of the first function result and its share of the second function result; the third computing participant obtains a third share of the first activation function result based on its share of the first function result and its share of the second function result; wherein, the combination of the first share of the first activation function result, the second share of the first activation function result and the third share of the first activation function result can restore the first activation function result.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the ciphertext of the first input is the result obtained by blinding the first input according to a first random number, the first computing participant blinds the share of the first input obtained by the first computing participant according to a second random number to obtain a first result, and sends the first result to the second computing participant, and the second random number is a share of the first random number; the second computing participant blinds the share of the first input obtained by the second computing participant according to a third random number to obtain a second result, and sends the second result to the first computing participant, and the third random number is another share of the first random number; the first computing participant and the second computing participant obtain the ciphertext of the first input according to the first result and the second result, respectively.

[0017] In combination with the first aspect, in some implementations of the first aspect, the first activation function includes a Relu activation function or a Sigmoid activation function.

[0018] In a second aspect, a computing device for an activation function in a neural network is provided, the device comprising: an acquisition module and a processing module, wherein the acquisition module is used for a first computing participant, a second computing participant and a third computing participant to respectively obtain shares of a first input, wherein the first input is input information of a first activation function, and the shares of the first input are shared among the first computing participant, the second computing participant and the third computing participant in a secret sharing manner; a processing module is used for the first computing participant and the second computing participant to respectively calculate a ciphertext of the first input according to their respective obtained shares of the first input; the processing module is also used for the first computing participant to determine a first function result according to the ciphertext of the first input and a first sub-function, wherein the first sub-function is a function obtained by splitting the first activation function in a function secret sharing manner. The processing module is further used for the second computing participant to determine the second function result according to the ciphertext of the first input and the second sub-function, wherein the second sub-function is another function share obtained by splitting the first activation function in a manner of secret sharing of the function, and the second function result is the output of the second sub-function; the processing module is further used for the first computing participant, the second computing participant and the third computing participant to respectively calculate the share of the first activation function result according to their respective shares of the first function result and their respective shares of the second function result, and the combination of the shares of the first activation function results respectively calculated by the first computing participant, the second computing participant and the third computing participant can restore the first activation function result, which is the output of the first activation function.

[0019] In combination with the second aspect, in certain implementations of the second aspect, the share of the first input is to share the first input among the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner.

[0020] In combination with the second aspect, in certain implementations of the second aspect, the processing module is also used by the third computing participant to split the first activation function in an offline phase by secret sharing of the function to obtain the first sub-function and the second sub-function; the sending module is used by the third computing participant to send the first sub-function and the second sub-function to the first computing participant and the second computing participant respectively.

[0021] In combination with the second aspect, in certain implementations of the second aspect, the processing module is also used by the first computing participant to share the first function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the first function result; the processing module is also used by the second computing participant to share the second function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the second function result.

[0022] In combination with the second aspect, in certain implementations of the second aspect, the first computing participant shares the first function result with the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner, and the second computing participant shares the second function result with the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner.

[0023] In combination with the second aspect, in certain implementations of the second aspect, the processing module is specifically used to: the first computing participant obtains the first share of the first activation function result based on the share of the first function result and the share of the second function result obtained by it; the second computing participant obtains the second share of the first activation function result based on the share of the first function result and the share of the second function result obtained by it; the third computing participant obtains the third share of the first activation function result based on the share of the first function result and the share of the second function result obtained by it; wherein, the combination of the first share of the first activation function result, the second share of the first activation function result and the third share of the first activation function result can restore the first activation function result.

[0024] In combination with the second aspect, in certain implementations of the second aspect, the ciphertext of the first input is the result obtained by blinding the first input according to the first random number, and the processing module is specifically used to: the first computing participant blinds the share of the first input obtained by the first computing participant according to the second random number to obtain a first result, and sends the first result to the second computing participant, and the second random number is a share of the first random number; the second computing participant blinds the share of the first input obtained by the second computing participant according to the third random number to obtain a second result, and sends the second result to the first computing participant, and the third random number is another share of the first random number; the first computing participant and the second computing participant obtain the ciphertext of the first input according to the first result and the second result, respectively.

[0025] In combination with the second aspect, in some implementations of the second aspect, the first activation function includes a Relu activation function or a Sigmoid activation function.

[0026] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0027] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0028] In a fourth aspect, a chip is provided, which obtains instructions and executes the instructions to implement the method in the above-mentioned first aspect and any implementation manner of the first aspect.

[0029] Optionally, as an implementation, the chip includes a processor and a data interface, and the processor reads instructions stored in the memory through the data interface to execute the method in the above-mentioned first aspect and any implementation of the first aspect.

[0030] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in the first aspect and any one of the implementation methods of the first aspect.

[0031] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any one of the implementations of the first aspect.

[0032] In a sixth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to the first aspect and any one of the implementations of the first aspect.

[0033] By way of example, these computer-readable storages include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0034] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic block diagram of the secure multi-party learning framework.

[0036] Figure 2 is a schematic diagram of the system architecture using the scenario of joint modeling between the financial industry and operators as an example.

[0037] FIG3 is a schematic flowchart of a method for calculating an activation function in a neural network provided in an embodiment of the present application.

[0038] FIG4 is a diagram showing the initial and final states of data held by various computing parties during the calculation of a ReLU function, as provided by an embodiment of the present application.

[0039] FIG5 is a schematic flowchart of a method for converting repetitive secret sharing into a distributed comparison function to solve the activation function ReLU, provided in an embodiment of the present application.

[0040] FIG6 is a schematic diagram of data randomly generated locally by a computing participant 3 in an offline phase, provided in an embodiment of the present application.

[0041] FIG7 is a diagram showing the initial and final states of data held by each computing party during the process of computing a Sigmoid function, as provided by an embodiment of the present application.

[0042] FIG8 is a schematic flowchart of a method for converting repetitive secret sharing into a distributed comparison function to solve the activation function Sigmoid, provided in an embodiment of the present application.

[0043] FIG9 is a schematic block diagram of a computing device 900 for an activation function in a neural network provided in an embodiment of the present application.

[0044] FIG10 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application.

[0045] FIG11 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0046] FIG12 is a schematic diagram of a network connection between computing devices 1500A and 1500B provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solution in this application will be described below with reference to the accompanying drawings.

[0048] This application will present various aspects, embodiments, or features around systems including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0049] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0050] In the embodiments of the present application, “corresponding” and “relevant” may sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0051] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person skilled in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0053] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0054] For the convenience of description, the relevant concepts involved in the embodiments of the present application are first described in detail below.

[0055] 1. Neural Networks

[0056] A neural network is a computer system that mimics the functionality of biological neural networks, learning and training from data to accomplish various tasks. It is a computational model composed of multiple nodes, each of which mimics the behavior of a biological neuron, receiving and processing input signals and generating output signals by learning and adjusting weights.

[0057] Neural networks typically consist of multiple layers of nodes, with the input layer receiving external data input, the output layer generating the final output, and the intermediate hidden layers processing and transforming the input. In a neural network, the connections between nodes have different weights, which can be adjusted through training and learning to improve the network's accuracy and performance. Neural networks are widely used in various fields, including image recognition, speech recognition, natural language processing, autonomous driving, medical diagnosis, and financial forecasting. Common neural network architectures include feedforward neural networks, convolutional neural networks, recurrent neural networks, and deep neural networks.

[0058] The computational operations during neural network model training are mainly composed of linear operations and nonlinear operations. Linear operations are mainly matrix multiplications, while nonlinear operations include various activation functions, such as Relu and Sigmoid.

[0059] 2. Secure multi-party learning

[0060] Secure multi-party learning is a distributed machine learning paradigm that allows multiple parties to collaboratively train AI models using their own data, without aggregating data from multiple parties (organizations or users). Traditional machine learning paradigms require the aggregation of large amounts of data for model training, and this training data is likely to come from multiple different organizations or users. Aggregating data from multiple different organizations or users poses a significant risk of data leakage, exposing information assets for organizations and potentially compromising personal privacy for individual users.

[0061] These issues pose significant challenges to AI model training. To address these challenges, secure multi-party learning (SML) technology has emerged. While ensuring data privacy, SML leverages secure multi-party computing techniques such as secret sharing and obfuscated circuits to collaboratively train AI models. This technology protects user data from multiple parties while leveraging their data to collaboratively train models, resulting in more robust models.

[0062] As an example, the secure multi-party learning framework is shown in Figure 1. This framework includes multiple data holders and computing participants. Neural network training in secure multi-party learning is divided into three specific steps: data sharing, model training, and model restoration. In Step 1, data sharing involves each data holder using secret sharing to share their data with each computing participant, ensuring that each computing participant possesses a secret share of the entire data set. In Step 2, during the training phase, computing participants interactively perform training on the entire data set, ultimately receiving a secret share of the trained neural network model. During this process, all intermediate computational results are also stored in the form of secret shares. In Step 3, during the model restoration phase, each computing participant transmits their secret share of the trained neural network model to a designated server, which then restores the final model.

[0063] 3. Secret Sharing

[0064] Secret sharing, also known as secret splitting, is a method of distributing a secret among a group of people. Each person receives a portion of the secret, called a share or shard. Only when enough shares are combined can the secret be recovered; each share is useless on its own. Secret sharing methods include, but are not limited to, additive secret sharing, repetitive secret sharing, and functional secret sharing.

[0065] 4. Additive secret sharing

[0066] Additive secret sharing is an instance of secret sharing that splits a secret value into multiple shares for sharing among multiple parties. In additive secret sharing, the original secret is represented as an integer and can be shared by splitting it into multiple parts. The original secret value can only be recovered when all the shares are reassembled. This method can be used to implement secure multi-party computation, such as data analysis while protecting individual privacy, or for secure computation in distributed computing.

[0067] One method for splitting additive secret sharing is to use field addition. For example, a secret value s can be split into two secret shares a and b, such that s = (a + b) mod p, where p is the ring size. The original secret value s can only be recovered by adding the secret shares a and b.

[0068] 5. Replicated secret sharing (RSS)

[0069] Recurring secret sharing (RSS) is another example of secret sharing, used to ensure that a secret value is shared among multiple entities and remains secure even if any of them fails or is attacked. In RSS, the original secret is split into multiple parts and distributed to multiple entities. Each entity has a copy of the secret, and the original secret value can only be recovered when at least a certain number of entities are present. This way, if one entity fails or is attacked, the other entities can still recover the original secret value using their respective copies.

[0070] For example, using repetitive secret sharing in a three-party scenario, a secret value s can be split into three secret shares a, b, and c, such that s = (a+b+c) mod p (where p is the size of the computation ring). The first party holds secret shares a and b, the second party holds secret shares b and c, and the third party holds secret shares a and c. The secret value s can only be recovered when any two of the three parties combine their secret shares.

[0071] 6. Function Secret Sharing

[0072] Function Secret Sharing (FSS) is a cryptographic concept and an instance of secret sharing, used to compute a secret function between multiple entities without leaking information about the function. In FSS, an original secret function is split into multiple parts and distributed to multiple entities. Each entity has a share of the secret function, and the original secret function can only be recovered when all shares of the secret function are reassembled.

[0073] One method for splitting a function secret sharing (FS) is to split it into two functions such that the sum of the two functions on the ring equals the original function. For example, split f(x) into f0(x) and f1(x) such that for any x, f(x) = f0(x) + f1(x).

[0074] Function secret sharing comes in two forms: a distributed point function and a distributed comparison function. A distributed comparison function compares data in a distributed system. The result of the comparison function serves as the output of the decision function. Therefore, the computational effort and complexity of the distributed comparison function are relatively low.

[0075] In a related technical solution, a secure multi-party learning framework includes two computing parties, one auxiliary computing party, and an unlimited number of data holders. Its security assumption is the semi-honest, honest majority security model—no party violates the protocol, and there is no collusion between the two computing parties, nor between the computing parties and the auxiliary computing party. The auxiliary computing party is responsible for generating multiplication triplets and function shares of function secret sharing in the offline phase and sending these multiplication triplets and function shares to the two computing parties. In the online phase, the two computing parties use the multiplication triplets generated offline and the secret shares split using additive secret sharing to perform matrix multiplication in the linear phase. They also use the function shares generated offline and the output of the matrix multiplication to calculate the activation function in the nonlinear phase. In this solution, matrix multiplication in the two-party additive secret sharing requires the use of multiplication triplets. Due to the large size of the multiplication triples, the online communication required for each matrix multiplication between the two computing parties is high, resulting in low efficiency.

[0076] In another related technical solution, a secure multi-party learning framework includes three computing participants. These three computing participants perform matrix multiplication operations in the linear phase through repetitive secret sharing. When performing the calculation of the activation function in the nonlinear phase, the nonlinear activation function is converted into a linear calculation, and the linear calculation is performed through repetitive secret sharing. In this solution, the matrix multiplication operation is implemented in the three-party repetitive secret sharing with the help of random numbers. Although the random number data is small, the online communication required by the three computing participants to perform a matrix multiplication is small. However, when performing the calculation of the activation function in the nonlinear phase, the nonlinear activation function is converted into a linear calculation. Due to the large amount of linear calculation, the overall efficiency is low.

[0077] In view of this, an embodiment of the present application provides a method for calculating the activation function in a neural network, which performs operations in the linear stage of the neural network through repetitive secret sharing, and performs operations in the nonlinear stage of the neural network through a combination of repetitive secret sharing and function secret sharing, thereby reducing the amount of communication and calculation required for neural network training.

[0078] It should be understood that the scenarios in which the method provided in the embodiments of the present application is applied may include but are not limited to: scenarios of cross-company or cross-industry data joint modeling, and no requirements are made on the specific data distribution method and the number of participants.

[0079] For example, in an internet advertising scenario, internet companies can leverage information from other internet companies or advertisers' purchasing data to build a model based on whether internet service users purchase related items, thereby improving advertising delivery efficiency.

[0080] As another example, financial industry users (such as banks and securities firms) can use data from other financial industry users or operators (such as China Mobile, China Telecom, China Unicom, etc.) and all their own data to jointly model and improve the accuracy of financial industry models (such as credit score models, credit card user default models, etc.), thereby helping to improve the overall operational efficiency and risk resistance of the financial industry.

[0081] For example, Figure 2 illustrates a system architecture for a joint modeling scenario involving the financial industry and carriers. As shown in Figure 2, the financial industry company and the carrier company each share data with three computing participants using a secret sharing mechanism. The three computing participants then perform interactive computations based on the data shared by the financial industry company and the carrier company. Each participant trains a model to obtain its own share, which they then send to the financial industry company. The financial industry company then restores the model based on its own share, ultimately yielding a restored neural network model.

[0082] Figure 3 is a schematic flow chart of a method for calculating an activation function in a neural network provided by an embodiment of the present application. As shown in Figure 3, the method may include steps 310-340, and steps 310-340 are described in detail below.

[0083] Step 310: The first computing participant, the second computing participant, and the third computing participant each obtain a share of the first input.

[0084] The first input is the input information of the first activation function, and the share of the first input is to share the first input among the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner.

[0085] It should be understood that the above-mentioned first activation function is the activation function in the nonlinear operation of the neural network model. The activation function can be a Relu activation function or a Sigmoid activation function. The embodiments of this application do not make specific limitations on this.

[0086] In an embodiment of the present application, the above-mentioned first input can be a data, or it can also be the result of performing matrix multiplication between the data and the neural network model. The embodiment of the present application does not make any specific limitations on this.

[0087] Step 320: The first computing participant determines the first function result based on the first input ciphertext and the first sub-function.

[0088] The first sub-function is a function share obtained by splitting the first activation function in a function secret sharing manner, the first function result is the output of the first sub-function, and the input of the first sub-function is the ciphertext of the first input.

[0089] The ciphertext of the first input may also be referred to as ciphertext. In one possible implementation, a blinding value may be added to the first input, and the result obtained after adding the blinding value to the first input is referred to as the ciphertext of the first input.

[0090] Step 330: The second computing participant determines the second function result based on the ciphertext of the first input and the second sub-function.

[0091] The second sub-function is another function share obtained by splitting the first activation function in a function secret sharing manner. The second function result is the output of the second sub-function, and the input of the second sub-function is the ciphertext of the first input.

[0092] Step 340: The first computing participant, the second computing participant, and the third computing participant respectively calculate the share of the first activation function result based on the share of the first function result and the share of the second function result.

[0093] The share of the first activation function result is the share obtained by splitting the first activation function result in the repetitive secret sharing manner, and the first activation function result is the output of the first activation function.

[0094] Specifically, in one possible implementation, the first computing participant obtains a first share of the first function result based on its share of the first function result and its share of the second function result; the second computing participant obtains a second share of the first function result based on its share of the first function result and its share of the second function result; the third computing participant obtains a third share of the first function result based on its share of the first function result and its share of the second function result; the combination of the first share, the second share and the third share can restore the first function result.

[0095] In an embodiment of the present application, the third computing participant can also split the first activation function into a first sub-function and a second sub-function by function secret sharing in the offline stage, and send the first sub-function and the second sub-function to the first computing participant and the second computing participant respectively.

[0096] In an embodiment of the present application, the first computing participant may also share the first function result with the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner, wherein the first computing participant, the second computing participant, and the third computing participant each obtain a share of the first function result.

[0097] In an embodiment of the present application, the second computing participant can also share the second function result with the first computing participant, the second computing participant and the third computing participant in a repetitive secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the second function result.

[0098] In the embodiment of the present application, if the first activation function is a Sigmoid activation function, the Sigmoid activation function can also be approximated as a piecewise comparison function according to a function approximation method.

[0099] In the above technical solution, when calculating nonlinear functions such as Relu and Sigmoid, the calculation is first converted to function secret sharing, and then converted back to repetitive secret sharing, which can reduce the communication and computation required for neural network training.

[0100] Taking the activation function as Relu as an example, the process of converting repetitive secret sharing into a distributed comparison function to solve the activation function Relu and finally converting it into repetitive secret sharing is described in detail. It should be understood that the following examples are only to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the application embodiments to the specific numerical values ​​or specific scenarios illustrated below. Those skilled in the art can obviously make various equivalent modifications or changes based on the examples given below, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0101] As an example, the expression of the activation function Relu is shown in formula (1).

[0102] As shown in Figure 4, the secret value x is shared with participants 1, 2, and 3 using repetitive secret sharing. Specifically, the shares in repetitive secret sharing include x1, x2, and x3. Initially, the three participants hold secret shares of the secret value x: participant 1 holds (x1, x2), participant 2 holds (x2, x3), and participant 3 holds (x3, x1). Next, we will calculate the ReLU function using function secret sharing without exposing any meaningful intermediate calculation results. Finally, participant 1 holds (y1, y2), participant 2 holds (y2, y3), and participant 3 holds (y3, y1).

[0103] It should be understood that y and x satisfy y = Relu(x), that is, y1 = Relu(x1), y2 = Relu(x2), y3 = Relu(x3), y1, y2 and y3 can be understood as the three secret shares of y, and the combination of y1, y2 and y3 can restore y.

[0104] FIG5 is a schematic flow chart of a method for converting repetitive secret sharing into a distributed comparison function to solve the activation function ReLU according to an embodiment of the present application. As shown in FIG5 , the method may include steps 510-560, which are described in detail below.

[0105] Step 510: Computing participant 3 randomly generates f, f1, f2, r, r1, r1 locally in the offline phase, and sends f1, r1, f2, r2 to computing participant 1 and computing participant 2.

[0106] As an example, as shown in FIG6 , in the offline phase, computing participant 3 randomly generates f, f1, f2, r, r1, r1 locally, where f, f1, f2 are functions and r, r1, r1 are random numbers.

[0107] The above data randomly generated locally by the computing participant 3 satisfies the relationship in the following formulas (2)-(4): f(x)=f1(x)+f2(x) (3) r=(r1+r2)mod p (4)

[0108] For example, computing participant 3 may send locally generated f1 and r1 to computing participant 1, and send f2 and r2 to computing participant 2.

[0109] Step 520: Calculation participant 1 and calculation participant 2 each locally add the share of x and the share of r to obtain the share of t, and then send it to the other party.

[0110] Take computing participant 1 as an example. Computing participant 1 locally calculates t1 = r1 + x1 and sends t1 to computing participant 2. It should be understood that t1 can be understood as a share of t, r1 can be understood as a share of r, and x1 can be understood as a share of x.

[0111] Take computing participant 2 as an example. Computing participant 2 locally calculates t2 = r2 + x2 + x3 and sends t2 to computing participant 1. It should be understood that t2 can be understood as another share of t, r2 can be understood as another share of r, and x2 and x3 can be understood as two other shares of x.

[0112] Step 530: Computing participant 1 and computing participant 2 restore t based on t1 and t2.

[0113] For computing participant 1, it calculates t1 locally and also receives t2 from computing participant 2. Therefore, computing participant 1 can calculate t=t1+t2, that is, t=r1+x1+r2+x2+x3=r+x.

[0114] For computing participant 2, it calculates t2 locally and also receives t1 from computing participant 1. Therefore, computing participant 2 can calculate t based on t=t1+t2, that is, t=r1+x1+r2+x2+x3=r+x.

[0115] It should be understood that t restored by computing participant 1 and computing participant 2 is the result of blinding the secret value x, or the result of encrypting the secret value x.

[0116] Step 540: Computing participant 1 uses t and f1 to perform function calculation to obtain g1, and shares g1 with computing participant 1, computing participant 2, and computing participant 3 respectively in a repetitive secret sharing manner.

[0117] As an example, computing participant 1 may substitute the calculated t into function f1, and the calculated result is g1, that is, g1=f1(t).

[0118] Computing participant 1 shares g1 with computing participant 1, computing participant 2, and computing participant 3 in a repetitive secret sharing manner. For example, computing participant 1 gets a portion of g1, computing participant 2 gets a portion of g1, and computing participant 3 gets a portion of g1.

[0119] Step 550: Computing participant 2 uses t and f2 to perform function calculation to obtain g2, and shares g2 with computing participant 1, computing participant 2, and computing participant 3 respectively in a repetitive secret sharing manner.

[0120] As an example, computing participant 2 may substitute the calculated t into function f2, and the calculated result is g2, that is, g2=f2(t).

[0121] Computing participant 2 shares g2 with computing participant 1, computing participant 2, and computing participant 3 in a repetitive secret sharing manner. For example, computing participant 1 gets a portion of g2, computing participant 2 gets a portion of g2, and computing participant 3 gets a portion of g2.

[0122] Step 560: The three calculation participants obtain the shares of g based on the shares of g1 and g2 respectively, and convert the shares of g into the shares of y.

[0123] In the embodiment of the present application, g can be understood as f. According to the above formula (2), if g is equal to 1, it means t <= r. Since t = x + r, it also means x <= 0. Otherwise, if g is equal to 0, it means t > r. Since t = x + r, it also means x > 0. Therefore, the conversion relationship between g and y = Relu (x) is shown in formula (5). [y] = (1-[g]) * [x] (5)

[0124] Where [y] represents the share of y, [g] represents the share of g, and [x] represents the share of x. The calculation method of [g] is shown in formula (6). [g]=[g1]+[g2] (6)

[0125] Here, [g1] represents the share of g1, and [g2] represents the share of g2.

[0126] As an example, computing participant 1 can calculate the share of y according to formula (6) and formula (5). Similarly, computing participant 2 can calculate the share of y according to formula (6) and formula (5), and computing participant 3 can calculate the share of y according to formula (6) and formula (5).

[0127] It should be understood that the shares of y calculated by one computing participant are y1 and y2 as shown in Figure 4, the shares of y calculated by two computing participants are y2 and y3 as shown in Figure 4, and the shares of y calculated by three computing participants are y3 and y1 as shown in Figure 4. Among them, the combination of y1, y2, and y3 can restore y, and the calculation result of Relu(x) can be obtained.

[0128] Below, taking the activation function as Sigmoid as an example, a detailed description is given of how to convert repetitive secret sharing into a distributed comparison function to solve the activation function Sigmoid, and finally convert it into a repetitive secret sharing process. It should be understood that the following examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the application embodiments to the specific numerical values ​​or specific scenarios illustrated below. Those skilled in the art can obviously make various equivalent modifications or changes based on the examples given below, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0129] As an example, the expression of the activation function Sigmoid is shown in formula (7).

[0130] As shown in Figure 7, the secret value x is shared with participants 1, 2, and 3 using repetitive secret sharing. Specifically, the shares in repetitive secret sharing include x1, x2, and x3. Initially, the three participants hold secret shares of the secret value x: participant 1 holds (x1, x2), participant 2 holds (x2, x3), and participant 3 holds (x3, x1). Next, we will calculate the Sigmoid function using function secret sharing without exposing any meaningful intermediate calculation results. Finally, participant 1 holds (y1, y2), participant 2 holds (y2, y3), and participant 3 holds (y3, y1).

[0131] It should be understood that y and x satisfy y = Sigmoid(x), that is, y1 = Sigmoid(x1), y2 = Sigmoid(x2), y3 = Sigmoid(x3), y1, y2 and y3 can be understood as the three secret shares of y, and the combination of y1, y2 and y3 can restore y.

[0132] In the embodiment of the present application, since Sigmoid is a point function rather than a comparison function, in order to implement Sigmoid using a distributed comparison function, a function approximation method is used to approximate the Sigmoid function with a piecewise function. That is, the Sigmoid function is approximated to the function shown in formula (8).

[0133] Figure 8 is a schematic flow chart of a method for converting repetitive secret sharing into a distributed comparison function to solve the activation function Sigmoid, provided by an embodiment of the present application. As shown in Figure 8, the method may include steps 810-860, which are described in detail below.

[0134] Step 810: Computing participant 3 randomly generates f, f1, f2, r, r1, r1 locally in the offline phase, and sends f1, r1, f2, r2 to computing participant 1 and computing participant 2.

[0135] Step 810 is the same as step 510 . Please refer to the description of step 510 for details.

[0136] Step 820: Computing participant 1 calculates u1 and v1 locally, and sends u1 and v1 to computing participant 2. Computing participant 2 calculates w locally, and sends w to computing participant 1.

[0137] As an example, computing participant 1 locally calculates u1=r1+x1+1 / 2, v1=r1+x1-1 / 2, and sends u1 and v1 to computing participant 2.

[0138] As an example, computing participant 2 calculates w=r2+x2+x3 locally and sends w to computing participant 1.

[0139] Step 830: Calculation participant 1 and calculation participant 2 calculate u and v.

[0140] For computing participant 1, it locally calculates u1 and v1 and receives w from computing participant 2. Therefore, computing participant 1 can calculate u = u1 + w, that is, u = r1 + x1 + 1 / 2 + r2 + x2 + x3 = r + x + 1 / 2. Computing participant 1 can also calculate v = u1 + w, that is, v = r1 + x1 - 1 / 2 + r2 + x2 + x3 = r + x - 1 / 2.

[0141] For computing participant 2, it calculates w locally and also receives u1 and v1 from computing participant 1. Therefore, computing participant 2 can calculate u = u1 + w, that is, u = r1 + x1 + 1 / 2 + r2 + x2 + x3 = r + x + 1 / 2. Computing participant 2 can also calculate v = u1 + w, that is, v = r1 + x1 - 1 / 2 + r2 + x2 + x3 = r + x - 1 / 2.

[0142] Step 840: Computing participant 1 uses u and f1 to perform function calculation to obtain g1, and uses v and f1 to perform function calculation to obtain h1, and then shares g1 and h1 with computing participant 1, computing participant 2, and computing participant 3 respectively in a repetitive secret sharing manner.

[0143] As an example, computing participant 1 may substitute the calculated u into function f1, and the calculated result is g1, that is, g1=f1(u).

[0144] Computing participant 1 shares g1 with computing participant 1, computing participant 2, and computing participant 3 in a repetitive secret sharing manner. For example, computing participant 1 gets a portion of g1, computing participant 2 gets a portion of g1, and computing participant 3 gets a portion of g1.

[0145] As an example, computing participant 1 may substitute the calculated v into function f1, and the calculated result is h1, that is, h1=f1(v).

[0146] Computing participant 1 shares h1 with computing participant 1, computing participant 2, and computing participant 3 in a repetitive secret sharing manner. For example, computing participant 1 gets a portion of h1, computing participant 2 gets a portion of h1, and computing participant 3 gets a portion of h1.

[0147] Step 850: Computing participant 2 uses u and f2 to perform function calculation to obtain g2, and uses v and f2 to perform function calculation to obtain h2, and shares g2 and h2 with computing participant 1, computing participant 2, and computing participant 3 respectively in a repetitive secret sharing manner.

[0148] As an example, computing participant 2 may substitute the calculated u into function f2, and the calculated result is g2, that is, g2=f2(u).

[0149] Computing participant 1 shares g2 with computing participant 1, computing participant 2, and computing participant 3 in a repetitive secret sharing manner. For example, computing participant 1 gets a portion of g2, computing participant 2 gets a portion of g2, and computing participant 3 gets a portion of g2.

[0150] As an example, computing participant 2 may substitute the calculated v into function f2, and the calculated result is h2, that is, h2=f2(v).

[0151] Computing participant 1 shares h2 with computing participant 1, computing participant 2, and computing participant 3 in a repetitive secret sharing manner. For example, computing participant 1 gets a portion of h2, computing participant 2 gets a portion of h2, and computing participant 3 gets a portion of h2.

[0152] Step 860: The three calculation participants respectively obtain the shares of g based on the shares of g1 and g2, obtain the shares of h based on the shares of h1 and h2, and convert the shares of g and h into shares of y.

[0153] In the embodiment of the present application, since h = x ≤ -1 / 2 and g = x ≤ 1 / 2, the conversion relationship between g, h and y = Sigmoid(x) is shown in formula (9). [y] = (1-[g])*[h]*[x] + (1-[h]) (9)

[0154] Where [y] represents the share of y, [g] represents the share of g, [x] represents the share of x, and [h] represents the share of h. The calculation method of [g] is shown in formula (6), and the calculation method of [h] is shown in formula (10). [h] = [h1] + [h2] (10)

[0155] Here, [h1] represents the share of h1, and [h2] represents the share of h2.

[0156] In the above technical solution, for the operation of the activation function in the nonlinear stage, by combining repetitive secret sharing and function secret sharing, since the calculation participant 3 did not participate in the calculation in the previous steps, the communication and calculation amount required for neural network training can be reduced.

[0157] The above describes in detail the method provided by the embodiment of the present application in conjunction with Figures 1 to 8. The following describes in detail the embodiment of the device of the present application in conjunction with Figures 9 to 12. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0158] Figure 9 is a schematic block diagram of a computing device 900 for an activation function in a neural network provided by an embodiment of the present application. The device 900 can be implemented by software, hardware, or a combination of both. The device 900 provided by an embodiment of the present application can implement the method flow shown in the embodiment of the present application, and the device 900 includes: an acquisition module 910, a processing module 920, wherein the acquisition module 910 is used for the first computing participant, the second computing participant, and the third computing participant to respectively obtain shares of the first input, wherein the first input is the input information of the first activation function, and the share of the first input is to share the first input among the first computing participant, the second computing participant, and the third computing participant in a secret sharing manner; the processing module 920 is used for the first computing participant and the second computing participant to respectively calculate the ciphertext of the first input according to their respective obtained shares of the first input; the processing module 920 is also used for the first computing participant to determine the first function result according to the ciphertext of the first input and the first sub-function, wherein the first sub-function is to perform a function secret sharing on the first activation function. A function share obtained after the split, the first function result is the output of the first sub-function; the processing module 920 is also used by the second computing participant to determine the second function result based on the ciphertext of the first input and the second sub-function, wherein the second sub-function is another function share obtained after the first activation function is split in a manner of secret sharing of the function, and the second function result is the output of the second sub-function; the processing module 920 is also used by the first computing participant, the second computing participant and the third computing participant to respectively calculate the share of the first activation function result based on their respective shares of the first function result and their respective shares of the second function result, and the combination of the shares of the first activation function results calculated by the first computing participant, the second computing participant and the third computing participant can restore the first activation function result, which is the output of the first activation function.

[0159] Optionally, the share of the first input is to share the first input among the first computing participant, the second computing participant, and the third computing participant in a repetitive secret sharing manner.

[0160] Optionally, the processing module 920 is also used by the third computing participant to split the first activation function in an offline phase by means of secret sharing of the function to obtain the first sub-function and the second sub-function; the sending module 930 is used by the third computing participant to send the first sub-function and the second sub-function to the first computing participant and the second computing participant respectively.

[0161] Optionally, the processing module 920 is also used by the first computing participant to share the first function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the first function result; the processing module 920 is also used by the second computing participant to share the second function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the second function result.

[0162] Optionally, the first computing participant shares the first function result with the first computing participant, the second computing participant and the third computing participant in a repetitive secret sharing manner, and the second computing participant shares the second function result with the first computing participant, the second computing participant and the third computing participant in a repetitive secret sharing manner.

[0163] Optionally, the processing module 920 is specifically used to: the first computing participant obtains a first share of the first activation function result based on its share of the first function result and its share of the second function result; the second computing participant obtains a second share of the first activation function result based on its share of the first function result and its share of the second function result; the third computing participant obtains a third share of the first activation function result based on its share of the first function result and its share of the second function result; wherein, the combination of the first share of the first activation function result, the second share of the first activation function result and the third share of the first activation function result can restore the first activation function result.

[0164] In combination with the second aspect, in certain implementations of the second aspect, the ciphertext of the first input is the result obtained by blinding the first input according to the first random number, and the processing module 920 is specifically used for the first computing participant to blind the share of the first input obtained by the first computing participant according to the second random number to obtain a first result, and send the first result to the second computing participant, and the second random number is a share of the first random number; the second computing participant to blind the share of the first input obtained by the second computing participant according to the third random number to obtain a second result, and send the second result to the first computing participant, and the third random number is another share of the first random number; the first computing participant and the second computing participant obtain the ciphertext of the first input according to the first result and the second result, respectively.

[0165] In combination with the second aspect, in some implementations of the second aspect, the first activation function includes a Relu activation function or a Sigmoid activation function.

[0166] The apparatus 900 herein may be embodied in the form of a functional module. The term "module" herein may be implemented in the form of software and / or hardware, and is not specifically limited thereto.

[0167] For example, a "module" can be a software program, hardware circuit, or a combination of the two that implements the aforementioned functions. For example, the implementation of acquisition module 910 will be described below, using acquisition module 910 as an example. Similarly, the implementation of other modules, such as processing module 920 and sending module 930, can refer to the implementation of acquisition module 910.

[0168] The acquisition module 910 is taken as an example of a software functional unit, and the acquisition module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing instance may be one or more. For example, the acquisition module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes one data center or multiple geographically close data centers. Generally, a region may include multiple AZs.

[0169] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0170] As an example of a hardware functional unit, acquisition module 910 may include at least one computing device, such as a server. Alternatively, acquisition module 910 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0171] The multiple computing devices included in acquisition module 910 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 910 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 910 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0172] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] It should be noted that: when the device provided in the above embodiment executes the above method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 910 can be used to execute any step in the above method, the processing module 920 can be used to execute any step in the above method, and the processing module 920 and the sending module 930 can be used to execute any step in the above method. The steps that the acquisition module 910, the processing module 920, and the sending module 930 are responsible for implementing can be specified as needed, and all the functions of the above device can be realized by respectively implementing different steps in the above method through the acquisition module 910, the processing module 920, and the sending module 930.

[0174] In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments above and will not be repeated here.

[0175] The method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and this application is not particularly limited, as long as the method provided in the embodiment of the present application can be used. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0176] 10 , a computing device provided in an embodiment of the present application is described in detail below.

[0177] FIG10 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 may be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in FIG10 includes: at least one processor 1510 and a memory 1520.

[0178] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0179] The processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the method provided in this application. Alternatively, the processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the functional modules provided in this application, thereby implementing the method provided in this application.

[0180] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0181] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520, and the communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540. For example, the processor 1510 can read and write data or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 10, but this does not mean that there is only one bus or one type of bus.

[0182] In one possible implementation, the processor 1510 is primarily responsible for interpreting computer program instructions (or codes) and processing data in the computer software. The computer program instructions and the data in the computer software may be stored in the memory 1520 or the cache 1516.

[0183] Optionally, the processor 1510 may be an integrated circuit chip having signal processing capabilities. By way of example and not limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0184] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514 .

[0185] Processing unit 1512, also known as a core, is the most important component of a processor. Processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All processor calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, leveraging parallel computing capabilities to accelerate program execution. Each processing unit has a fixed logical structure. For example, a processing unit includes logical units such as a level 1 cache, a level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0186] In one implementation example, the memory control unit 1514 is configured to control data exchange between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to the memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0187] In one implementation example, each memory control unit 1514 addresses the memory 1520 via a system bus. An arbiter (not shown in FIG. 10 ) is configured in the system bus to handle and coordinate competing accesses by multiple processing units 1512 .

[0188] In an implementation example, the processing unit 1512 and the memory control unit 1514 are connected to each other via connection lines inside the chip, such as address lines, so as to achieve communication between the processing unit 1512 and the memory control unit 1514.

[0189] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When a processing unit 1512 needs to read data, it first searches the cache for the required data. If the data is found, it executes the request directly; if not, it searches the memory. Because the cache runs much faster than the memory, the cache helps the processing unit 1512 run faster.

[0190] Memory 1520 can provide runtime space for processes in computing device 1500. For example, memory 1520 stores computer programs (specifically, program code) used to generate processes. After the computer program is executed by the processor to generate a process, the processor allocates corresponding storage space for the process in memory 1520. Furthermore, the aforementioned storage space further includes a text segment, an initialized data segment, a bit-initialized data segment, a stack segment, a heap segment, and the like. Memory 1520 stores data generated during the execution of the process, such as intermediate data or process data, in the storage space corresponding to the aforementioned process.

[0191] Optionally, the memory is also called the internal memory. Its function is to temporarily store the data processed by the processor 1510 and the data exchanged with external storage such as a hard disk. As long as the computer is running, the processor 1510 will load the data to be calculated into the internal memory for calculation. When the calculation is completed, the processing unit 1512 will transmit the result.

[0192] By way of example and not limitation, memory 1520 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0193] The structure of the computing device 1500 listed above is only an example, and the present application is not limited thereto. The computing device 1500 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk storage, etc. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 10.

[0194] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0195] As shown in Figure 11, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0196] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions of the above method.

[0197] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster can store different instructions, each for performing part of the functions of the above-mentioned apparatus. In other words, the instructions stored in the memory 1520 in different computing devices 1500 can implement the functions of one or more modules in the above-mentioned apparatus.

[0198] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN), a local area network (LAN), or the like. FIG12 illustrates one possible implementation. As shown in FIG12 , two computing devices 1500A and 1500B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0199] It should be understood that the functionality of the computing device 1500A shown in FIG12 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0200] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to implement the functions of the apparatus provided above.

[0201] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be executed on a computing device cluster or stored in any available medium. When executed by a computing device cluster, the computer program product causes the computing device cluster to perform the above-mentioned method or implement the functions of the above-mentioned apparatus.

[0202] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method provided above.

[0203] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method provided above.

[0204] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0205] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0206] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0208] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0209] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0210] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0211] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for calculating an activation function in a neural network, characterized in that: The method comprises: The first computing participant, the second computing participant, and the third computing participant respectively obtain shares of the first input, wherein the first input is input information of the first activation function, and the shares of the first input are the first input shared among the first computing participant, the second computing participant, and the third computing participant in a secret sharing manner; The first computing participant and the second computing participant respectively calculate the ciphertext of the first input according to the shares of the first input obtained by each party; The first computing participant determines a first function result according to the ciphertext of the first input and a first sub-function, wherein the first sub-function is a function share obtained by splitting the first activation function in a function secret sharing manner, and the first function result is an output of the first sub-function; The second computing participant determines a second function result according to the ciphertext of the first input and a second sub-function, wherein the second sub-function is another function share obtained by splitting the first activation function in a manner of function secret sharing, and the second function result is an output of the second sub-function; The first computing participant, the second computing participant and the third computing participant respectively calculate the shares of the first activation function result according to their respective shares of the first function result and their respective shares of the second function result. The combination of the shares of the first activation function results calculated by the first computing participant, the second computing participant and the third computing participant can restore the first activation function result, which is the output of the first activation function.

2. The method according to claim 1, characterized in that The method further comprises: The third computing participant splits the first activation function in an offline phase by means of the function secret sharing to obtain the first sub-function and the second sub-function; The third computing participant sends the first sub-function and the second sub-function to the first computing participant and the second computing participant respectively.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The first computing participant shares the first function result with the first computing participant, the second computing participant, and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant, and the third computing participant each obtain a share of the first function result; The second computing participant shares the second function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtains a share of the second function result.

4. The method according to any one of claims 1 to 3, characterized in that The first computing participant, the second computing participant, and the third computing participant calculate the share of the first activation function result according to the share of the first function result and the share of the second function result, respectively, including: The first computing participant obtains a first share of the first activation function result according to the share of the first function result and the share of the second function result obtained by the first computing participant; The second computing participant obtains a second share of the first activation function result according to the share of the first function result and the share of the second function result obtained by the second computing participant; The third computing participant obtains a third share of the first activation function result according to its share of the first function result and its share of the second function result; The combination of the first share of the first activation function result, the second share of the first activation function result, and the third share of the first activation function result can restore the first activation function result.

5. The method according to any one of claims 1 to 4, characterized in that The ciphertext of the first input is a result obtained by blinding the first input according to the first random number. The first computing participant and the second computing participant respectively calculate the ciphertext of the first input according to the shares of the first input obtained by each party, including: The first computing participant blinds the share of the first input obtained by the first computing participant according to a second random number to obtain a first result, and sends the first result to the second computing participant, where the second random number is a fraction of the first random number. shares; The second computing participant blinds the share of the first input obtained by the second computing participant according to a third random number to obtain a second result, and sends the second result to the first computing participant, where the third random number is another share of the first random number; The first computing participant and the second computing participant obtain the ciphertext of the first input according to the first result and the second result respectively.

6. The method according to any one of claims 1 to 5, characterized in that The first activation function includes a Relu activation function or a Sigmoid activation function.

7. A computing device for an activation function in a neural network, characterized in that: The device comprises: An acquisition module is used for the first computing participant, the second computing participant, and the third computing participant to respectively obtain shares of the first input, wherein the first input is input information of the first activation function, and the share of the first input is to share the first input among the first computing participant, the second computing participant, and the third computing participant in a secret sharing manner; A processing module, configured for the first computing participant and the second computing participant to respectively calculate the ciphertext of the first input according to the shares of the first input obtained by each party; The processing module is further used for the first computing participant to determine a first function result according to the first input ciphertext and a first sub-function, wherein the first sub-function is a function share obtained by splitting the first activation function in a function secret sharing manner, and the first function result is an output of the first sub-function; The processing module is further used for the second computing participant to determine a second function result according to the ciphertext of the first input and a second sub-function, wherein the second sub-function is another function share obtained by splitting the first activation function in a manner of function secret sharing, and the second function result is an output of the second sub-function; The processing module is also used for the first computing participant, the second computing participant and the third computing participant to respectively calculate the shares of the first activation function result according to their respective shares of the first function result and their respective shares of the second function result. The combination of the shares of the first activation function results calculated by the first computing participant, the second computing participant and the third computing participant can restore the first activation function result, and the first activation function result is the output of the first activation function.

8. The device according to claim 7, characterized in that The processing module is further used for the third computing participant to split the first activation function in an offline phase in a manner of function secret sharing to obtain the first sub-function and the second sub-function; A sending module is used for the third computing participant to send the first sub-function and the second sub-function to the first computing participant and the second computing participant respectively.

9. The method according to claim 7 or 8, characterized in that: The processing module is further configured for the first computing participant to share the first function result with the first computing participant, the second computing participant, and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant, and the third computing participant each obtain a share of the first function result; The processing module is also used for the second computing participant to share the second function result with the first computing participant, the second computing participant and the third computing participant in a secret sharing manner, wherein the first computing participant, the second computing participant and the third computing participant each obtain a share of the second function result.

10. The device according to any one of claims 7 to 9, characterized in that The processing module is specifically used for: The first computing participant obtains a first share of the first activation function result according to the share of the first function result and the share of the second function result obtained by the first computing participant; The second computing participant obtains a second share of the first activation function result according to the share of the first function result and the share of the second function result obtained by the second computing participant; The third computing participant obtains a third share of the first activation function result according to its share of the first function result and its share of the second function result; The combination of the first share of the first activation function result, the second share of the first activation function result, and the third share of the first activation function result can restore the first activation function result.

11. The device according to any one of claims 7 to 10, characterized in that The ciphertext of the first input is a result obtained by blinding the first input according to the first random number, and the processing module is specifically used for: The first computing participant blinds the share of the first input obtained by the first computing participant according to a second random number, Obtaining a first result, and sending the first result to the second computing participant, wherein the second random number is a share of the first random number; The second computing participant blinds the share of the first input obtained by the second computing participant according to a third random number to obtain a second result, and sends the second result to the first computing participant, where the third random number is another share of the first random number; The first computing participant and the second computing participant obtain the ciphertext of the first input according to the first result and the second result respectively.

12. The device according to any one of claims 7 to 11, characterized in that The first activation function includes a Relu activation function or a Sigmoid activation function.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Backdoor attack detection and identification method for privacy protection neural network model

    CN112989438A

  • Privacy protection multi-party reinforcement learning system and method based on Shamir security multi-party computing

    CN114118368A

  • Neural network prediction system for privacy protection

    CN115065463A

  • Calculation method and device for activation function in neural network and calculation equipment

    CN117114059A

  • Secret sigmoid function calculation system, secret logistic regression calculation system, secret sigmoid function calculation apparatus, secret logistic regression calculation apparatus, secret sigmoid function calculation method, secret logistic regression calculation method, and program

    US20220006614A1