A method, device and apparatus for joint deployment of neural networks by multiple parties

Abstract

One or more embodiments of the present specification provide a method for multiple parties to jointly deploy a neural network. The neural network is used for text processing. One of the multiple parties obtains the input data distribution of the first nonlinear layer of the neural network, and then determines a piecewise polynomial function whose fitting error meets preset conditions. Among them, the fitting error is: the weighted cumulative value of the difference between the output results of each input value passing through the first nonlinear function and the piecewise polynomial function, with the distribution probability value of the input value in the data distribution as the weight. Finally, according to the piecewise polynomial function, the deployment method of the first nonlinear layer in the multiple parties is determined. Through the above method, the influence of the input data distribution is taken into account when fitting the function, so as to reduce the error of the fitted piecewise polynomial function by controlling the fitting error. In addition, while ensuring that the error is not much different from the method of the related art, the size of the highest power and the number of segment segments can be reduced, thereby reducing the computational overhead of the neural network.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of multi-party secure computing technology, and more particularly, to a method, apparatus, and device for jointly deploying a neural network by multiple parties. Background Art

[0002] Processing text through neural networks to achieve text question answering, text translation, etc. has become a widely used technology. With the development of neural network technology, in some scenarios, multiple parties need to collaborate to use neural networks to complete reasoning to ensure data security and privacy.

[0003] In order to protect data security, multiple parties in the existing technology generally use privacy computing methods to complete the reasoning of neural networks. Privacy computing is a method that achieves collaborative computing among multiple parties while ensuring the privacy data security of each party. Common privacy computing methods include homomorphic encryption (HE) and secure multi-party computing (MPC). Whether it is homomorphic encryption or MPC, secret state operations generally only support linear operations such as multiplication and addition. However, there are generally a large number of nonlinear function operations in neural networks, such as normalized exponential function (softmax).

[0004] Therefore, we hope to have an improved solution to better deploy neural networks including nonlinear functions in scenarios of multi-party joint reasoning. Summary of the invention

[0005] In view of this, one or more embodiments of this specification provide a method, apparatus, medium, and device for multiple parties to jointly deploy a neural network.

[0006] According to a first aspect of one or more embodiments of this specification, a method for multiple parties to jointly deploy a neural network is proposed, wherein the neural network is used for text processing, and the method is executed by any one of the multiple parties, including:

[0007] Obtaining input data distribution formed by the text data set in the first nonlinear layer of the neural network; the first nonlinear layer corresponds to the first nonlinear function;

[0008] Determine a piecewise polynomial function whose fitting error satisfies a preset condition, wherein the fitting error is a weighted accumulation of error values ​​of each input value within the definition domain, wherein the error value of any input value is determined according to the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, and the weighted weight value is a distribution probability value of the input value in the input data distribution;

[0009] According to the piecewise polynomial function, a distribution deployment manner of the first nonlinear layer in the multiple parties is determined.

[0010] According to a second aspect of one or more embodiments of the present specification, a device for jointly deploying a neural network by multiple parties is provided, wherein the neural network is used for text processing, and the device is deployed in execution by any one of the multiple parties, and comprises: an input data distribution acquisition module, which is used to acquire the input data distribution formed by the text data set in the first nonlinear layer of the neural network; the first nonlinear layer corresponds to a first nonlinear function;

[0011] A piecewise polynomial function determination module is used to determine a piecewise polynomial function whose fitting error satisfies a preset condition, wherein the fitting error is a weighted accumulation of error values ​​of each input value within a definition domain, wherein the error value of any input value is determined according to the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, and the weighted weight value is a distribution probability value of the input value in the input data distribution;

[0012] The first nonlinear layer deployment module is used to determine the distribution deployment mode of the first nonlinear layer in the multiple parties according to the piecewise polynomial function.

[0013] According to a third aspect of an embodiment of this specification, a computer device is provided, comprising: a processor; a memory for storing processor executable instructions; the processor executes the executable instructions to implement the multi-party joint deployment of a neural network as described in the first aspect of an embodiment of this specification.

[0014] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, which, when executed by a processor, implements the multi-party joint deployment of a neural network as described in the first aspect of the embodiments of this specification.

[0015] In one or more embodiments of the present specification, a method for jointly deploying a neural network by multiple parties is proposed. The neural network is used for text processing. One of the multiple parties obtains the input data distribution of the first nonlinear layer of the neural network, and then determines a piecewise polynomial function whose fitting error meets a preset condition. Among them, the fitting error is: the weighted cumulative value of the difference between the output results of each input value passing through the first nonlinear function and the piecewise polynomial function, with the distribution probability value of the input value in the data distribution as the weight. Finally, the deployment method of the first nonlinear layer in the multiple parties is determined according to the piecewise polynomial function.

[0016] Through the above method, the influence of input data distribution is taken into account when fitting the function, so that by controlling the fitting error, compared with the piecewise polynomial function fitted in the related art, under the same highest power and number of segmented segments as the piecewise polynomial function of the related art, the function fitted by the method of this specification has a smaller average error for the same batch of data. In addition, while ensuring that the error is not much different from the method of the related art, the size of the highest power and the number of segmented segments can be reduced, the computational overhead of the neural network can be reduced, and the computational efficiency of the neural network deployed by multiple parties can be improved.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the specification, and together with the description, serve to explain the principles of the specification;

[0019] Figure 1 is a flowchart of a method for jointly deploying a neural network by multiple parties according to an exemplary embodiment of this specification;

[0020] Figure 2 is a schematic diagram of input data distribution according to an exemplary embodiment of this specification;

[0021] Figure 3 is a schematic diagram of a GELU function shown in this specification according to an exemplary embodiment;

[0022] Figure 4 is a block diagram of a device for jointly deploying a neural network by multiple parties according to an exemplary embodiment of this specification;

[0023] Figure 5 It is a hardware structure diagram of a computer device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0026] When multiple participants jointly reason through neural networks, it may involve confidential nonlinear function operations. For example, participant A has a user question text constructed based on personal privacy data of a certain user, and participant B has a neural network that can answer the question. Participant A wants to use participant B's neural network to infer the answer to the question while ensuring that the question text is not leaked; participant B also needs to ensure that the parameters of its own neural network are not leaked.

[0027] In this case, Party A can use privacy computing methods to conduct joint reasoning with Party B. Common privacy computing methods include HE and MPC.

[0028] There are many nonlinear operations in neural networks, such as activation functions such as softmax. However, HE and MPC generally only support linear operations such as dense multiplication and addition. Therefore, in order to complete the reasoning of neural networks, when deploying neural networks in MPC scenarios, it is necessary to convert nonlinear functions into approximate polynomial functions. A polynomial function refers to a function obtained by multiplying and adding a constant and an independent variable x a finite number of times. For example, a polynomial function can be , where n is a positive integer, is any real number, and the value range of i is 1-n.

[0029] It can be seen that the polynomial function only includes multiplication and addition operations between the input value x and other values, which meets the requirements of dense state operations. In addition, since the definition domain of nonlinear functions is relatively wide, it is generally impossible to fit a polynomial function that is similar to the nonlinear function at all positions in the definition domain. Therefore, the nonlinear function is generally converted into an approximate piecewise polynomial function.

[0030] In the related art, a nonlinear function is generally fitted as a piecewise polynomial function with a large number of segments and a high highest power. For example, the function expression of GELU in the related art is shown in formula (1), and in the related art, the exponential function (exponential, or exponential activation function) and the Gaussian Error Linear Unit (GELU) function can be fitted as piecewise polynomial functions shown in the following formulas (2) and (3).

[0031] GELU ( x ) = 0 . 5 × x ( 1 + tanh [ 2 π ( x + 0 . 044715 x 3 ) ] ) (1)

[0032] (2)

[0033] (3)

[0034] Among them, formula (2) is the function obtained by fitting the exponential function, and formula (3) is the function obtained by fitting the GELU function. is a polynomial function with a highest power of 3, is a polynomial function with a maximum power of 6.

[0035] When there are many segments in the piecewise polynomial function, it is necessary to perform multiple operations: comparing the input ciphertext data with the segment threshold. In addition, when the highest power of any segment of the piecewise polynomial is high, it will be necessary to perform multiple ciphertext multiplications. Both the ciphertext comparison operation and the ciphertext multiplication operation will consume a lot of computational overhead. If you want to reduce the computational overhead, you need to reduce the number of segments or reduce the size of the highest power in the piecewise polynomial.

[0036] However, in the related art, when converting a nonlinear function into an approximate linear function or a polynomial function, it is restricted by the calculation accuracy and it is impossible to further reduce the number of segments or the size of the highest power.

[0037] Specifically, the relevant technology generally fits the nonlinear function and the piecewise polynomial function based on the mean square error in the definition domain. In practical applications, more attention is paid to whether the calculation results of the nonlinear function and the fitted piecewise polynomial function for the same batch of data are similar. However, the input data may not be evenly distributed in the definition domain, which may make the piecewise polynomial function as a whole similar to the nonlinear function, but the error between the two in a certain value is slightly larger. When the value accounts for a large proportion of the input data, the error between the output of the piecewise polynomial function and the nonlinear function for a batch of data is large.

[0038] It can be seen that the related technologies are restricted by the calculation accuracy and cannot reduce the size of the highest power and the number of segments, which leads to low calculation efficiency of the neural network jointly deployed by multiple parties.

[0039] In order to solve the above problems, the method of this specification proposes a method for jointly deploying a neural network by multiple parties. The neural network is used for text processing. One of the multiple parties obtains the input data distribution of the first nonlinear layer of the neural network, and then determines a piecewise polynomial function whose fitting error meets the preset conditions. Among them, the fitting error is: the weighted cumulative value of the difference between the output results of each input value passing through the first nonlinear function and the piecewise polynomial function, with the distribution probability value of the input value in the data distribution as the weight. Finally, according to the piecewise polynomial function, the deployment method of the first nonlinear layer in the multiple parties is determined.

[0040] Through the above method, the influence of input data distribution is taken into account when fitting the function, so that by controlling the fitting error, compared with the piecewise polynomial function fitted in the related art, under the same highest power and number of segmented segments as the piecewise polynomial function of the related art, the function fitted by the method of this specification has a smaller average error for the same batch of data. In addition, while ensuring that the error is not much different from the method of the related art, the size of the highest power and the number of segmented segments can be reduced, the computational overhead of the neural network can be reduced, and the computational efficiency of the neural network deployed by multiple parties can be improved.

[0041] Next, a method for multi-party joint deployment of a neural network provided in this specification will be described. The neural network of this specification can be used for text processing. Of course, in other optional implementations, the neural network of this specification can also be used for other processing, such as classification of high-risk users, etc.

[0042] In an optional implementation, the neural network may be a Transformer neural network, which may achieve better results in text processing.

[0043] In a multi-party joint deployment scenario, the multiple parties can use the neural network model for joint reasoning in an MPC environment. The method can be performed by any of the multiple parties. In an optional embodiment, the method can also be performed by the party that owns the neural network.

[0044] like Figure 1 As shown, Figure 1 This is a flowchart of a method for jointly deploying a neural network by multiple parties according to an exemplary embodiment of the present specification, including:

[0045] Step 101, obtaining the input data distribution formed by the text data set in the first nonlinear layer of the neural network.

[0046] The first nonlinear layer corresponds to a first nonlinear function.

[0047] The first nonlinear layer corresponds to a first nonlinear function, which may be GELU, an exponential function, or softmax, etc. This specification does not limit the specific form of the first nonlinear function.

[0048] The input data distribution can be the input data of the first nonlinear layer, that is, the distribution probability function of the data input to the first nonlinear layer. Figure 2 As shown, Figure 2 Figure (a) is a possible distribution histogram of the input data of the GELU function. Figure 2 Figure (b) in the figure is a possible distribution bar graph of the input data of the exponential function. The horizontal axis in the figure is the size of the input value, and the vertical axis is the distribution probability value corresponding to the input value. It should be noted that Figure 2 (a) Figure 2 (b) does not represent a certain characteristic of the input data of the two functions, but the input data distribution of a nonlinear layer of the neural network determined according to a specific text dataset. The input data distribution may be different when the text dataset or the depth of the nonlinear layer is different. Figure 2 (a) and Figure 2 The form shown in (b) is different.

[0049] As for the method of obtaining the input data distribution, the input data distribution can be obtained by the executor through a text data set (such as training data) stored by itself. In addition, the input data distribution can also be obtained through interaction with another participant. The implementation method of step 101 will be described below through several optional implementation methods.

[0050] In an optional embodiment, the method can be executed by a party that owns the model, and step 101 can be: obtaining a subset of the training data as a text data set, inputting the text data set into the neural network model, and determining the input data distribution based on the input formed by the text data set in the first nonlinear layer of the neural network model.

[0051] In an optional implementation manner, in step 101 , the input data distribution may be determined by receiving input data sent by another party.

[0052] For example, step 101 may include: receiving an input data subset sent by another party, wherein the input data subset is input data of the first nonlinear layer obtained by the other party inputting its own text data into the neural network; aggregating the input data subsets sent by each other party and the input data subset of the party into an input data set; and obtaining the input data distribution according to the input data set by statistics.

[0053] Specifically, other parties may send input data subsets to the execution party, where the input data subsets include input data of the first nonlinear function. The execution party aggregates the input data subsets to obtain input data distribution.

[0054] For another example, step 101 may include: receiving the data scale and data sub-distribution sent by the other party, wherein the data sub-distribution is obtained by the other party inputting its own text data into the neural network to obtain an input data subset of the first non-linear layer, and performing statistics on the input data subset; aggregating the data scales and data sub-distributions of each party to obtain the input data distribution.

[0055] Receive the input data sub-distribution and data scale of the first nonlinear layer obtained by each participant based on its own stored data. The data scale refers to the input data distribution obtained based on the amount of data. The executor determines the proportion of the input data of each participant based on the data scale of each participant, summarizes the data sub-distribution of each participant, and calculates the weighted average of the data sub-distribution with the calculated proportion as the weight to obtain the above input data distribution.

[0056] It should be noted that in the above two examples, in order to ensure data privacy, the input data subset or data sub-distribution received here can be obtained by other parties inputting their desensitized data into some layers of the model. In the case where the original input data of the model cannot be inferred from the input data subset or data sub-distribution, the input data subset can also be obtained by other parties based on local private text data.

[0057] For another example, step 101 may include: obtaining text data sets from multiple parties; inputting each text data in the text data set into the neural network respectively to obtain each input data of each text data in the first nonlinear layer; and obtaining the input data distribution according to the value of each input data.

[0058] Specifically, the difference from the previous method is that the original text dataset input into the neural network is obtained in this method. The executor can obtain the text datasets of each participant, and obtain the input dataset of the first nonlinear layer by inputting the text datasets of each participant into the neural network. Then, the input dataset is statistically analyzed to obtain the input data distribution. In order to ensure data privacy and security, the text dataset that can be received here can be the desensitized data held by other parties.

[0059] Of course, the above embodiments do not limit the present specification. Any method that can obtain the input data distribution can be used as an implementation method of step 101 in the present specification. For example, a public text data set can be obtained and the input data distribution can be determined based on the public text data set.

[0060] Step 103: Determine a piecewise polynomial function whose fitting error satisfies a preset condition.

[0061] Among them, the fitting error is the weighted accumulation of the error values ​​of each input value in the definition domain, wherein the error value of any input value is determined according to the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, and the weighted weight value is the distribution probability value of the input value in the input data distribution.

[0062] Specifically, in step 103, when fitting, the total error between the fitted piecewise polynomial function and the first nonlinear function is determined according to the probability of the input value as a weight. Thus, when fitting the function, the input value with a high probability has a greater impact on the fitting, so that the output of the fitted piecewise polynomial function for the input value with a high probability is closer to the output of the first nonlinear function for the input value. In turn, the error performance of the fitted polynomial function on a batch of data is better.

[0063] A piecewise polynomial function is a function that consists of multiple segments, each of which is composed of a polynomial function, a linear function, or a constant function.

[0064] The fitting error refers to the error between the fitted piecewise polynomial function and the first nonlinear function, which is specifically obtained by weighted accumulation of the error values ​​of each input value. The error in the related art is generally obtained by the average value of the error of each input value. In order to introduce the input data distribution into the calculation of the fitting error, the distribution probability value in the input data distribution is used as the weighted weight.

[0065] The error value may be: the absolute value of the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, or the square of the difference or the norm of the difference, etc. This specification does not limit this.

[0066] In an optional implementation, when the error value is determined based on the square of the difference value, for the fitted piecewise polynomial function, the fitting error can be calculated according to the following formula (4):

[0067] ∫ l h p ( x )[ f ( x ) − f '( x )] 2 dx (4)

[0068] in, is the fitted piecewise polynomial function, is the first nonlinear function, is the input data distribution, x is the input data, l and h are The upper and lower bounds of the domain of .

[0069] The preset condition may be that the fitting error is minimal, or the fitting error is less than a preset threshold, etc.

[0070] In an optional embodiment, when the preset condition is that the fitting error is minimized, the goal to be achieved in step 103 can be as shown in formula (5):

[0071] min f '( x ) ∫ l h p ( x )[ f ( x ) − f '( x )] 2 dx (5)

[0072] As for the specific implementation method of step 103, it can be: according to the initial values ​​of N demarcation points, obtain the candidate value of each demarcation point; for the candidate segmentation method corresponding to any candidate value combination, determine the candidate polynomial function of each segment with the minimum fitting error under the candidate segmentation method, and the minimum fitting error; determine the piecewise polynomial function according to the minimum fitting error and the candidate polynomial function under each candidate segmentation method.

[0073] Specifically, the initial values ​​of multiple cutoff points can be obtained, and then the candidate values ​​of multiple cutoff points can be determined based on the initial values. For example, there are two initial values ​​of the cutoff points at the beginning, which are 1 and -1, and then the candidate values ​​of multiple cutoff points can be determined based on the initial values ​​of the two cutoff points. For example, for the initial value 1, three candidate values ​​are determined, which are 1.1, 1 and 0.9; for the initial value -1, three candidate values ​​are also determined, which are -1.1, -1 and -0.9.

[0074] The initial value of the demarcation point can be preset, or it can be: the initial values ​​of N demarcation points are determined according to the input data distribution. For the latter specific implementation method, for example, the inflection point of the input data distribution can be searched, and the inflection point of the input data distribution can be used as the initial value of the demarcation point; or it can be: searching for a point where the probability value of the input data distribution takes a specific value, and using the point as the initial value of the demarcation point.

[0075] The candidate value of the demarcation point may be determined according to a preset range and step size. Specifically, the process of determining the candidate value may be: for each of the initial values ​​of the N demarcation points, within the preset range of the initial value, obtain the candidate value of the demarcation point according to the step size.

[0076] For example, the preset range is 0.1 around the initial value, and the step size is 0.05. If the initial value is 1, several candidate values ​​of 1, 1.05, 1.1, 0.95 and 0.9 can be determined.

[0077] In addition, in order to ensure that the fitted piecewise polynomial does not bring a lot of calculation after deployment, the highest power of the polynomial function of each segment of the piecewise polynomial function can be preset.

[0078] When the initial value of the demarcation point and the highest power of the polynomial function of each segment are preset, their values ​​can be determined based on factors such as input data distribution and the curvature change of the first nonlinear function.

[0079] For example, within a certain input value range, the distribution probability value of the input value is low, and the curve of the first nonlinear function within this range is relatively smooth. In this case, the probability mutation position can be used as the initial value of the dividing point, and the highest power of this function segment can be set to 1.

[0080] In addition, for the selection of the highest power of the curve, since the magnitude of the fitting error corresponding to different highest powers is similar, the initial value can be used as the segmentation threshold in advance, and multiple highest powers can be set for each segment. Simple fitting can be performed under multiple highest powers to determine the highest power whose magnitude of the fitting error meets the requirements. In this way, the fitted function can achieve a balance between the amount of calculation and accuracy.

[0081] After determining the candidates corresponding to multiple dividing points, multiple candidate value combinations can be determined. For example, there are two dividing points, and the candidate values ​​of the two dividing points are 1.1, 1 and 0.9, and -1.1, -1 and -0.9. Then the following candidate value combinations can be determined: (1.1,-1.1), (1.1,-1), (1.1,-0.9), (1,-1.1), (1,-1), (1,-0.9), (0.9,-1.1), (0.9,-1) and (0.9,-0.9). Each bracket represents a candidate value combination. Each candidate value combination also corresponds to a candidate segmentation method, in which the candidate value combination is divided into multiple segments.

[0082] Then, the candidate polynomial function with the smallest fitting error for each segment under each candidate value combination can be determined by function fitting, and the fitting errors of each candidate polynomial function can be summed to obtain the minimum fitting error under the candidate segmentation method.

[0083] For ease of understanding, it should be noted that the candidate polynomial function refers to a polynomial function with the smallest fitting error in a segment. The piecewise polynomial function refers to a piecewise function including a plurality of segmented candidate polynomial functions.

[0084] Further, according to the minimum fitting error determined under each candidate segmentation mode, a piecewise polynomial function that meets the preset conditions can be selected. For example, if the preset condition is that the fitting error is the minimum, then the minimum minimum fitting error can be determined from the minimum fitting errors corresponding to each candidate segmentation mode, and the piecewise polynomial function composed of the candidate polynomial functions of each segment corresponding to the error is used as the final piecewise polynomial function.

[0085] Step 105: Determine the distribution deployment mode of the first nonlinear layer in the multiple parties according to the piecewise polynomial function.

[0086] After the piecewise polynomial function is determined in step 103, it is then possible to determine how to deploy the first nonlinear layer in the multi-party.

[0087] For example, it can be determined on which side the model is deployed. When deployed in the side, the first nonlinear function of the first nonlinear layer can be replaced with the piecewise polynomial function for deployment.

[0088] For the method using HE, after the piecewise polynomial function corresponding to the first nonlinearity is determined, the operator corresponding to the piecewise polynomial function may also be determined.

[0089] For MPC, after determining the piecewise polynomial function corresponding to the first nonlinear layer, the model can be split by secret sharing and other methods, such as splitting the coefficients of the piecewise polynomial function into multiple shards and distributing them to multiple participants.

[0090] In addition, before deploying the first nonlinear layer, the neural network may be fine-tuned so that the neural network can ensure the accuracy of the inference results even when using a certain piecewise polynomial function for calculation.

[0091] In addition, a neural network may include multiple nonlinear layers, and multiple nonlinear layers may correspond to the same nonlinear function. For different nonlinear layers, due to their different depths in the neural network, the input data of different nonlinear layers have different data distributions. Therefore, in order to ensure the accuracy of the fitted piecewise polynomial function, even if multiple nonlinear layers correspond to the same nonlinear function, the above method should be executed for different nonlinear layers to determine the piecewise polynomial functions corresponding to different nonlinear layers.

[0092] In other words, the neural network further includes a second nonlinear layer, the second nonlinear layer corresponds to the first nonlinear function, and the piecewise polynomial functions determined for the first nonlinear layer and the second nonlinear layer are different.

[0093] In addition, for different participants, since the text data of different participants are almost the same in data distribution, when multiple participants use the same neural network model, it is not necessary to perform fitting of the first linear layer for different participants respectively.

[0094] Next, the method of jointly deploying a neural network by multiple parties will be described through a specific embodiment.

[0095] First, the initial values ​​of multiple cutoff points are determined. The influence of the input data distribution and the curvature change of the nonlinear function are considered when determining the initial values ​​of the cutoff points.

[0096] For example, for Figure 2 A possible distribution of the Exponential function shown in FIG. 1 . In the related art, the segmentation threshold of the Exponential is set to -14, and Figure 2 The probability of the Exponential function appearing at a value less than -4 is low, and the curve is relatively smooth, so a linear function can be used for fitting. In this way, since the segmentation threshold is set to the right, the segmentation domain on the right side of the segmentation threshold becomes smaller, and compared with related technologies, a lower-order polynomial can be used to fit the right-side segment. For example, a cubic polynomial can be selected to fit the right-side domain instead of using a Taylor series expansion containing higher-order terms.

[0097] For the GELU function, the curve of the GELU function is as follows Figure 3 As shown in the figure, in the related art, for this function, two segments are divided between [-4,3] in the definition domain, and the two segments use a cubic polynomial and a sixth polynomial respectively.

[0098] In the method of this specification, since the probability of occurrence of values ​​greater than 0 is low, even if the difference between the fitted polynomial function and the nonlinear function in this segment is large, it does not affect the size of the fitting error of a batch of data as a whole. For the case where the input value is less than -2.1, this part can be simply regarded as equal to 0. For the middle segment [T1, T2] = [-2.1, 0.2], the GELU function in this segment has more fluctuations and a higher probability of occurrence, and a cubic polynomial can be assigned to fit this part. In this way, for the GELU function, the initial values ​​of the two dividing points can be determined, which are -2.1 and 0.2 respectively.

[0099] For each segment, the random coefficients of the polynomial function of the segment may be initialized according to the determined highest power.

[0100] Second, piecewise polynomial fitting can be performed. Here we take the example of only one dividing point as an example.

[0101] 1. Generate a candidate value set S of the demarcation point according to the initial value T of the demarcation point, which can be generated according to a preset range and step size. The preset range and step size can be preset according to requirements for fitting error, calculation efficiency, etc.

[0102] 2. For each candidate value in the candidate value set S, determine the segmentation method corresponding to each candidate value. Then, for each segmentation method, fit the candidate polynomial function in each segment.

[0103] The specific fitting and the method for determining the minimum fitting error can be seen in the above description.

[0104] Third, the deployment mode of the nonlinear layer is determined according to the fitted piecewise polynomial function.

[0105] Through the above method, the piecewise polynomial function similar to the nonlinear function can have a lower degree and fewer segments. For example, the highest degree of the Exponential function is reduced to 3, and the division in the Taylor series is no longer required. The GELU is changed from four segments to three segments, and the highest degree is changed from 6 to 3. The piecewise polynomial functions fitted by the Exponential function and the GELU function are shown in the following formulas (6) and (7), respectively:

[0106] (6)

[0107] (7)

[0108] Wherein, formula (6) is the piecewise polynomial function corresponding to the Exponential function, is a value in the candidate value set near -14. Formula (7) is the piecewise polynomial function corresponding to the GELU function. is a value in the candidate value set near -2.1, is a value in the set of candidate values ​​near 0.2. represents the polynomial function with the highest power of 3. Correspondingly, is a polynomial function whose highest power is 1.

[0109] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0110] like Figure 4 As shown, Figure 4 This is a block diagram of a device for jointly deploying a neural network by multiple parties according to an exemplary embodiment of this specification, wherein the neural network is used for text processing and the device is deployed in any one of the multiple parties. The device includes:

[0111] An input data distribution acquisition module 410 is used to acquire the input data distribution formed by the text data set in the first nonlinear layer of the neural network; the first nonlinear layer corresponds to the first nonlinear function;

[0112] A piecewise polynomial function determination module 420 is used to determine a piecewise polynomial function whose fitting error satisfies a preset condition, wherein the fitting error is a weighted accumulation of error values ​​of each input value within the definition domain, wherein the error value of any input value is determined according to the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, and the weighted weight value is a distribution probability value of the input value in the input data distribution;

[0113] The first nonlinear layer deployment module 430 is used to determine the distribution deployment mode of the first nonlinear layer in the multiple parties according to the piecewise polynomial function.

[0114] In an optional implementation, the input data distribution acquisition module 410 is specifically used to: receive an input data subset sent by the other party, where the input data subset is the input data of the first nonlinear layer obtained by the other party inputting its own private text data into the neural network; aggregate the input data subsets sent by each other party and the input data subset of the party into an input data set; and obtain the input data distribution based on the input data set.

[0115] In an optional implementation, the input data distribution acquisition module 410 is specifically used to: obtain text data sets from multiple parties; input each text data in the text data set into the neural network respectively to obtain each input data of each text data in the first non-linear layer; and obtain the input data distribution according to the value of each input data.

[0116] In an optional implementation manner, the piecewise polynomial function determination module 420 specifically includes:

[0117] A candidate value acquisition submodule is used to acquire a candidate value for each cutoff point according to the initial values ​​of the N cutoff points;

[0118] A minimum fitting error determination submodule is used to determine, for a candidate segmentation method corresponding to any candidate value combination, a candidate polynomial function of each segment with the minimum fitting error under the candidate segmentation method, and the minimum fitting error;

[0119] The piecewise polynomial function determination submodule is used to determine the piecewise polynomial function according to the minimum fitting error and the candidate polynomial function under each candidate segmentation mode.

[0120] In an optional implementation, the initial values ​​of the N demarcation points are determined based on the input data distribution.

[0121] In an optional implementation, the candidate value acquisition submodule is specifically configured to acquire, for each of the initial values ​​of the N demarcation points, a candidate value of the demarcation point according to a step size within a preset range of the initial value.

[0122] In an optional implementation, the first nonlinear function is a Gaussian error linear unit GELU, an exponential function, or a normalized exponential function softmax.

[0123] In an optional implementation, the neural network further includes a second nonlinear layer, the second nonlinear layer corresponds to the first nonlinear function, and the polynomial functions determined for the first nonlinear layer and the second nonlinear layer are different.

[0124] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0125] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0126] like Figure 5 As shown, Figure 5 A hardware structure diagram of a computer device in which the apparatus for jointly deploying a neural network by multiple parties in an embodiment is located is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050.

[0127] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0128] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0129] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0130] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0131] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0132] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0133] The embodiments of this specification also provide a computer program product, which, when executed by a processor, implements the above-mentioned method of multi-party joint deployment of a neural network.

[0134] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method of multi-party joint deployment of a neural network.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A method for multiple parties to jointly deploy a neural network, the neural network being used for text processing, the method being executed by any one of the multiple parties, comprising: Obtaining input data distribution formed by the text data set in the first nonlinear layer of the neural network; The first nonlinear layer corresponds to a first nonlinear function; Determine a piecewise polynomial function whose fitting error satisfies a preset condition, wherein the fitting error is a weighted accumulation of error values ​​of each input value within the definition domain, wherein the error value of any input value is determined according to the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, and the weighted weight value is a distribution probability value of the input value in the input data distribution; Determining, according to the piecewise polynomial function, a distribution deployment mode of the first nonlinear layer in the multiple parties; The step of determining a piecewise polynomial function whose fitting error satisfies a preset condition comprises: According to the initial values ​​of N dividing points, a candidate value of each dividing point is obtained; For a candidate segmentation method corresponding to any candidate value combination, determine a candidate polynomial function of each segment with the minimum fitting error under the candidate segmentation method, and the minimum fitting error; The piecewise polynomial function is determined according to the minimum fitting error and the candidate polynomial function under each candidate segmentation mode.

2. The method according to claim 1, wherein: The step of obtaining the data distribution probability formed by the text data set in the first nonlinear layer of the neural network includes: Obtain text datasets from multiple sources; Inputting each text data in the text data set into the neural network respectively to obtain each input data of each text data in the first nonlinear layer; The input data distribution is obtained by statistics according to the value of each input data.

3. The method according to claim 1, wherein: The step of obtaining the data distribution probability formed by the text data set in the first nonlinear layer of the neural network includes: Receiving an input data subset sent by the other party, wherein the input data subset is input data of the first nonlinear layer obtained by the other party inputting the private text data owned by the other party into the neural network; Aggregate the input data subsets sent by each other party and the input data subset of the party into an input data set; The input data distribution is obtained by statistics according to the input data set.

4. The method according to claim 1, wherein: The initial values ​​of the N demarcation points are determined according to the input data distribution.

5. The method according to claim 1, wherein obtaining a candidate value of each cutoff point according to the initial values ​​of the N cutoff points comprises: For each of the initial values ​​of the N demarcation points, a candidate value of the demarcation point is obtained according to a step size within a preset range of the initial value.

6. The method according to claim 1, wherein: The first nonlinear function is a Gaussian error linear unit GELU, an exponential function, or a normalized exponential function softmax.

7. The method according to claim 1, wherein: The neural network also includes a second nonlinear layer, the second nonlinear layer corresponds to the first nonlinear function, and the piecewise polynomial functions determined for the first nonlinear layer and the second nonlinear layer are different.

8. A device for jointly deploying a neural network by multiple parties, wherein the neural network is used for text processing, the device being deployed in any one of the multiple parties, comprising: An input data distribution acquisition module, used to acquire the input data distribution formed by the text data set in the first nonlinear layer of the neural network; The first nonlinear layer corresponds to a first nonlinear function; A piecewise polynomial function determination module is used to obtain a candidate value for each demarcation point according to the initial values ​​of the N demarcation points; for a candidate segmentation method corresponding to any candidate value combination, determine the candidate polynomial function of each segment with the minimum fitting error under the candidate segmentation method, and the minimum fitting error; determine the piecewise polynomial function according to the minimum fitting error and the candidate polynomial function under each candidate segmentation method, wherein the fitting error is a weighted accumulation of the error values ​​of each input value within the definition domain, wherein the error value of any input value is determined according to the difference between the output results of the first nonlinear function and the piecewise polynomial function for the input value, and the weighted weight value is the distribution probability value of the input value in the input data distribution; The first nonlinear layer deployment module is used to determine the distribution deployment mode of the first nonlinear layer in the multiple parties according to the piecewise polynomial function.

9. A computer program product, which, when executed by a processor, implements the method for multi-party joint deployment of a neural network as described in any one of claims 1 to 7.

10. A computer device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method for multi-party joint deployment of a neural network as described in any one of claims 1 to 7 by running the executable instructions.

Images