Method and apparatus for jointly training a logistic regression model

Abstract

Embodiments of the present specification provide a method for jointly training a logistic regression model by multiple parties, each of which holds partial data forming a training sample set, the method being applied to any first party among the multiple parties; the method involves multiple rounds of training, wherein any round includes: obtaining multiple batches divided from the training sample set in the round; for each batch of the multiple batches, determining a first loss slice based on a first feature slice and a first label slice of the batch of training samples and a first parameter slice of the logistic regression model, the first loss slice being used to restore a training loss corresponding to the batch; in a case where the first party is a designated party, determining a training loss in the round based on multiple training losses corresponding to the multiple batches, the training loss in the round being used to evaluate a convergence state of the logistic regression model after training in the round.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of machine learning, and in particular to a method and device for jointly training a logistic regression model. BACKGROUND

[0002] Data is the basis of machine learning, and in most industries, due to privacy and security issues, data often exists in the form of an island, and even the integration of data between different departments of the same company also faces many obstacles. In view of the dilemma of data islands and data privacy, the industry proposes federated learning (FL), also known as federated machine learning, joint learning, or alliance learning. FL is a machine learning framework that aims to help multiple data parties to use data and build machine learning models under the premise of meeting data privacy and security.

[0003] The logistic regression (LR) model is a widely used machine learning model, and training the LR model under the FL framework is a hot research topic. However, there is currently no solution to safely evaluate the convergence degree of the LR model during training, which limits the training efficiency of the LR model.

[0004] Therefore, there is a need for a joint training solution for the LR model that protects the privacy and security of each party's input while efficiently evaluating the convergence state of the LR model to guide the subsequent process of LR model training, such as stopping training in time to avoid overfitting. SUMMARY

[0005] One or more embodiments of the present specification describe a method and device for jointly training a logistic regression model, which evaluates the convergence state of the logistic regression model in a timely and efficient manner to guide the subsequent training process, thereby training a logistic regression model with excellent performance.

[0006] According to a first aspect, a method for jointly training a logistic regression model by multiple parties is provided, each party holding part of the data forming a training sample set, and the method is applied to any first party among the multiple parties. The method involves multiple training periods, and any period includes: obtaining multiple batches divided for the training sample set in the current period; for each batch in the multiple batches, determining a first loss slice based on a first feature slice and a first label slice of the batch of training samples and a first parameter slice of the logistic regression model, the first loss slice being used to restore the training loss corresponding to the batch; in the case that the first party is a designated party, determining a current training loss based on multiple training losses corresponding to the multiple batches, the current training loss being used to evaluate the convergence state of the logistic regression model after the current training.

[0007] In an embodiment, the method further comprises: determining a first gradient slice based on the first feature slice, the first label slice and the first parameter slice; and updating the first parameter slice using the first gradient slice.

[0008] In an embodiment, before determining the current training loss based on the plurality of training losses corresponding to the plurality of batches, the method further comprises: receiving, from other parties in the plurality of parties except the first party, other loss slices corresponding to the plurality of batches; and restoring a corresponding training loss based on the first loss slice and the other loss slices corresponding to each batch.

[0009] In an embodiment, determining the current training loss based on the plurality of training losses corresponding to the plurality of batches comprises: randomly sampling a predetermined number of training losses from the plurality of training losses; and calculating an average of the predetermined number of training losses as the current training loss.

[0010] In an embodiment, after determining the current training loss, the method further comprises: determining the convergence state according to the current training loss; and stopping training and determining the logistic regression model after the current training as the trained logistic regression model in a case where the convergence state is converged.

[0011] In a specific embodiment, the current training is not the first training; and wherein determining the convergence state according to the current training loss comprises: determining a change rate of training loss according to the current training loss and a last training loss; and determining the convergence state as converged in a case where the change rate is less than a preset threshold.

[0012] In another specific embodiment, the method further comprises: stopping training and prompting a staff to adjust a hyperparameter used for training in a case where the convergence state is abnormal, the hyperparameter comprising a learning rate or a sample amount per batch.

[0013] According to a second aspect, there is provided a method for jointly training a logistic regression model by a plurality of parties, each of which holds part of data forming a training sample set, the method being applied to any first party in the plurality of parties; the method involves a plurality of training periods, each of which comprises: obtaining a plurality of batches divided from the training sample set in a current training period; determining, for each batch in the plurality of batches, a first gradient slice based on a first feature slice and a first label slice of the batch of training samples and a first parameter slice of the logistic regression model, the first gradient slice being used to update the first parameter slice and restore a corresponding training gradient; and in a case where the first party is a designated party, determining a training gradient norm of the current training period based on a plurality of training gradients corresponding to the plurality of batches, the training gradient norm being used to evaluate a convergence state of the logistic regression model after the current training.

[0014] In an embodiment, before determining the training gradient norm of the current epoch based on the plurality of training gradients corresponding to the plurality of batches, the method further comprises: receiving, from other parties in the plurality of parties except the first party, other gradient shards corresponding to the batches; and restoring the training gradient corresponding to each batch based on the first gradient shard and the other gradient shard corresponding to the batch.

[0015] In an embodiment, determining the training gradient norm of the current epoch based on the plurality of training gradients corresponding to the plurality of batches comprises: randomly sampling a predetermined number of training gradients from the plurality of training gradients; calculating a norm of each of the predetermined number of training gradients; and obtaining a mean of the norms as the training gradient norm.

[0016] In an embodiment, after determining the training gradient norm of the current epoch, the method further comprises: determining the convergence state according to the training gradient norm; and in a case where the convergence state is converged, stopping the training and determining the logistic regression model after the training of the current epoch as the trained logistic regression model.

[0017] In a specific embodiment, the current epoch is a non-first epoch; and determining the convergence state according to the training gradient norm of the current epoch comprises: determining a change rate of the training gradient norm according to the training gradient norm of the current epoch and a training gradient norm of a previous epoch; and in a case where the change rate is less than a preset threshold, determining the convergence state as converged.

[0018] In another specific embodiment, the method further comprises: in a case where the convergence state is abnormal, stopping the training and prompting a staff to adjust a hyperparameter used in the training, the hyperparameter comprising a learning rate or a sample amount per batch.

[0019] According to a third aspect, a method for training a logistic regression model in a multi-party manner is provided, the plurality of parties each holding part of data forming a training sample set, the method being applied to any first party in the plurality of parties; the method involving multi-epoch training, wherein any epoch comprises: obtaining a plurality of batches divided from the training sample set for the current epoch; performing multi-round iterative update on a first parameter shard of the logistic regression model by using a plurality of sets of first training feature shards and first training label shards corresponding to the plurality of batches, to obtain an updated first parameter shard of the current epoch; determining a first validation loss shard based on a first validation feature shard and a first validation label shard corresponding to a validation sample set and the updated first parameter shard of the current epoch, for restoring a validation loss of the current epoch; and in a case where the first party is a designated party, evaluating a convergence state of the logistic regression model after the training of the current epoch based on the validation loss of the current epoch.

[0020] In an embodiment, before evaluating the convergence state of the trained logistic regression model of the current period based on the current period validation loss, the method further comprises: receiving other validation loss shards from other parties in the plurality of parties except the first party; and restoring the current period validation loss based on the first validation loss shard and the other validation loss shards.

[0021] In an embodiment, the validation sample set is pre-divided into a plurality of validation subsets; wherein determining the first validation loss shard based on the first validation feature shard and the first validation label shard corresponding to the validation sample set and the updated first parameter shard of the current period comprises: determining a current period validation subset corresponding to the current period training from the plurality of validation subsets; and determining the first validation loss shard based on the first validation feature shard and the first validation label shard of the current period validation subset and the updated first parameter shard of the current period; wherein the method further comprises: updating the updated first parameter shard of the current period according to the first validation feature shard and the first validation label shard.

[0022] In an embodiment, the current period training is not the first period; wherein evaluating the convergence state of the trained logistic regression model of the current period based on the current period validation loss comprises: determining a change rate of the validation loss according to the current period validation loss and a last period validation loss; and determining that the convergence state is converged in a case that the change rate is less than a preset threshold.

[0023] In an embodiment, after evaluating the convergence state of the trained logistic regression model of the current period based on the current period validation loss, the method further comprises: stopping training and determining the trained logistic regression model of the current period as a trained logistic regression model in a case that the convergence state is converged; or stopping training and prompting a staff to adjust a hyperparameter used for training in a case that the convergence state is abnormal, the hyperparameter comprising a learning rate or a sample amount per batch.

[0024] According to a fourth aspect, there is provided an apparatus for jointly training a logistic regression model by a plurality of parties, each of which holds partial data forming a training sample set, the apparatus being integrated in a first party of the plurality of parties; the apparatus is configured to perform any one of a plurality of training periods involved in the training by: a batch obtaining unit configured to obtain a plurality of batches of the training sample set for the current training period; a loss slice determining unit configured to determine, for each batch of the plurality of batches, a first loss slice based on a first feature slice and a first label slice of the training samples of the batch and a first parameter slice of the logistic regression model, the first loss slice being used to restore a training loss corresponding to the batch; and a training loss determining unit configured to determine, in a case that the first party is a designated party, a training loss of the current training period based on a plurality of training losses corresponding to the plurality of batches, the training loss of the current training period being used to evaluate a convergence state of the logistic regression model after the current training period.

[0025] According to a fifth aspect, there is provided an apparatus for jointly training a logistic regression model by a plurality of parties, each of which holds partial data forming a training sample set, the apparatus being integrated in a first party of the plurality of parties; the apparatus is configured to perform any one of a plurality of training periods involved in the training by: a batch obtaining unit configured to obtain a plurality of batches of the training sample set for the current training period; a gradient slice determining unit configured to determine, for each batch of the plurality of batches, a first gradient slice based on a first feature slice and a first label slice of the training samples of the batch and a first parameter slice of the logistic regression model, the first gradient slice being used to update the first parameter slice and restore a training gradient corresponding to the batch; and a gradient norm determining unit configured to determine, in a case that the first party is a designated party, a training gradient norm of the current training period based on a plurality of training gradients corresponding to the plurality of batches, the training gradient norm of the current training period being used to evaluate a convergence state of the logistic regression model after the current training period.

[0026] According to a sixth aspect, there is provided an apparatus for jointly training a logistic regression model by a plurality of parties, each of which holds partial data forming a training sample set, the apparatus being applied to a first party of the plurality of parties; the apparatus is configured to perform a plurality of training periods, wherein any one of the training periods comprises: a batch obtaining unit configured to obtain a plurality of batches of the training sample set for the current training period; a parameter slice determining unit configured to perform a plurality of rounds of iterative updating on a first parameter slice of the logistic regression model by using a plurality of groups of first training feature slices and first training label slices corresponding to the plurality of batches, to obtain an updated first parameter slice of the current training period; a loss slice determining unit configured to determine a first validation loss slice based on a first validation feature slice and a first validation label slice of a validation sample set and the updated first parameter slice of the current training period, the first validation loss slice being used to restore a validation loss of the current training period; and a convergence state evaluating unit configured to evaluate, in a case that the first party is a designated party, a convergence state of the logistic regression model after the current training period based on the validation loss of the current training period.

[0027] According to a seventh aspect, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of the first aspect or the second aspect or the third aspect.

[0028] According to an eighth aspect, there is provided a computing device comprising a memory having stored therein executable code and a processor that, when executing the executable code, implements the method of the first aspect or the second aspect or the third aspect.

[0029] With the method and device provided by the embodiments of the present specification, the convergence state of the logistic regression model is evaluated in a timely and efficient manner to guide the subsequent training process, so as to train a logistic regression model with excellent performance. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, on the premise of not creating labor, can also obtain other drawings according to these drawings.

[0031] Figure 1 The communication architecture diagram of the multi-party joint training of the logistic regression model according to one embodiment is shown;

[0032] Figure 2 The method flow diagram of the multi-party joint training of the logistic regression model according to one embodiment is shown;

[0033] Figure 3 The scene diagram of data vertical partitioning and data horizontal partitioning is shown;

[0034] Figure 4 The interaction diagram of the two-party cooperative execution of the SMM according to one embodiment is shown;

[0035] Figure 5 The method flow diagram of the multi-party joint training of the logistic regression model according to another embodiment is shown;

[0036] Figure 6 The method flow diagram of the multi-party joint training of the logistic regression model according to another embodiment is shown;

[0037] Figure 7 The device structure diagram of the multi-party joint training of the logistic regression model according to one embodiment is shown;

[0038] Figure 8 The device structure diagram of the multi-party joint training of the logistic regression model according to another embodiment is shown;

[0039] Figure 9 A schematic diagram of the apparatus structure for multi-party joint training of a logistic regression model according to yet another embodiment is shown. Detailed Implementation

[0040] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0041] As mentioned earlier, training LR models within the FL framework is a major research focus. During LR model training, as the model parameters gradually converge, a common scenario is that after the model parameters reach a certain level of complexity, the training error on the training dataset gradually decreases, but the error on the validation set increases with continued training. This is overfitting; that is, although the model's complexity increases, the model cannot make good predictions on datasets other than the training set. Excessive training not only consumes resources but also reduces the effectiveness of model training, so overfitting must be avoided. In another possible scenario, the model's error on the validation set fluctuates and fails to stabilize, preventing the model parameters from converging to the optimal solution.

[0042] To avoid these undesirable situations, it is necessary to determine the convergence of the model during training. However, in the training of privacy-preserving LR models, it is necessary to protect the private inputs of the participants. Usually, after each party has trained to completion, they exchange parameter slices to build a complete model before the training effect can be seen. Since there are no intermediate output results, it is difficult to determine the convergence of the model during training.

[0043] Based on the above observations and analysis, the inventors propose a scheme for multi-party joint training of LR models, which can securely characterize the convergence of the model during the training process while protecting the data privacy of the participants.

[0044] Figure 1 This diagram illustrates the communication architecture of a multi-party joint training logistic regression model according to one embodiment. Figure 1 As shown, during the training process, any i-th party P among the multiple parties... i In addition to fragmentation based on the features it holds [x] i And tag-based fragmentation [y] i Update the parameter fragment [θ] it holds. i ( Figure 1 The diagram shows an update to [θ′]. i In addition to the above, the corresponding loss slices [L] are also calculated. i Therefore, when the i-th party belongs to the designated party, loss fragmentation is performed based on the multiple parties. wherein N denotes the total number of participants, the corresponding training loss L is reduced, and convergence is judged according to the training loss L. In this way, the safety of the convergence can be safely characterized.

[0045] The steps of the scheme for implementing the above inventive concept are described below in combination with specific embodiments.

[0046] Figure 2 A method flowchart for training a logistic regression model in cooperation of multiple parties is shown, wherein each party holds part of the data forming the training sample set. Typically, the training data is distributed in different forms in the multiple parties, including horizontal split scenarios and vertical split scenarios. Figure 3 The data split forms in two scenarios are shown in the case of two parties in the above table, Figure 3 The left side of the above table is a horizontal split scenario of data, i.e., the two parties have different samples, but the feature items and label styles of these samples are the same. For example, there are two banks, both of which have credit features and credit labels of users, but the user groups are different. Figure 3 The right side of the above table is a vertical split scenario of data, i.e., the two parties have the same business objects, but they have different features of the same business objects, and only one party has the label. For example, two platforms have the same set of users, but the user features maintained by them are different, and only one party has the user label.

[0047] The above describes the distribution of the training sample set in the multiple parties. On the other hand, the business objects involved in the training sample set can be users, goods, or events, etc. For example, the events can be access events, login events, transaction events, or complaint events, etc. In addition, the training sample includes the object features and object labels of the corresponding business objects. In one embodiment, the business object is a user, and accordingly, the user features can include attribute features and behavior features of the user. For example, the attribute features include age, gender, residence, interest, etc., and the behavior features include transaction frequency, login frequency, social activity, number of credit service usage, default record, etc. The user label can be a high-risk user or a low-risk user. In another embodiment, the business object is a good, and accordingly, the good features can include good name, category, place of origin, cost, selling price, etc., and the good label can be a popular product or a cold product. In yet another embodiment, the business object is an event, and accordingly, the event features can include time, geographic location, network address, and related terminal device information, etc. of the event occurrence, and the event label can be a security event or an abnormal event. The content of the training sample is introduced here.

[0048] It should be understood that the training of the LR model includes multiple epochs, and the complete traversal of the training sample set during the training process is referred to as an epoch. In addition, each of the above-mentioned parties (or multiple participants, multiple data parties, multiple computing parties) can be implemented as a device, a server or a cluster of devices with computing and processing capabilities.

[0049] Figure 2 The method shown can be performed by any one of the above-mentioned parties (or referred to as the first party, P i party) executes the steps of any one of the multiple epochs involved in the method performed by the first party: Figure 2 The steps of any one of the multiple epochs involved in the method performed by the first party are shown in the following:

[0050] In step S210, the multiple batches divided for the training sample set in the current epoch are obtained. In step S220, for each batch in the multiple batches, a first loss slice is determined based on the first feature slice and the first label slice of the batch of training samples and the first parameter slice of the logistic regression model, which is used to restore the training loss corresponding to the batch. In step S230, in the case where the first party is a designated party, the training loss in the current epoch is determined based on the multiple training losses corresponding to the multiple batches, which is used to evaluate the convergence state of the logistic regression model after the training in the current epoch.

[0051] The above steps are described as follows:

[0052] First, in step S210, the multiple batches divided for the training sample set in the current epoch are obtained. It should be understood that the amount of data in the training set is usually large, so it needs to be divided into multiple batches for training; and the division method of the training set is usually different in each epoch.

[0053] Since the batch information for dividing the training sample set does not cause privacy leakage, the determination and propagation of the batch information can be flexibly set. In an embodiment, a certain party in the multiple parties can divide the batches according to the numbers of the samples in the training sample set, and establish a mapping relationship between the training epochs and the batch division information, wherein the batch division information includes the batches to which the numbers of the samples belong; then, the designated party can broadcast this mapping relationship to the other parties. In this way, each party can obtain this mapping relationship. Therefore, in this step, the first party can determine the batch division information corresponding to the current epoch according to the obtained mapping relationship. In another embodiment, the multiple parties can take turns to divide the batches, and accordingly, in this step, the first party can obtain the batch information from the party responsible for the batch division in the current epoch.

[0054] The above can obtain a plurality of batches corresponding to the current training. Then, in step S220, for each batch of the plurality of batches, based on the first feature slice and the first label slice of the training sample of the batch, and the first parameter slice of the logistic regression model, a first loss slice is determined for restoring the training loss corresponding to the batch.

[0055] It should be noted that the above first feature slice and first label slice can be obtained based on secret sharing (SS) technology. SS technology is a basic technology for secure computing. After the original data is randomly split and distributed, each distributed data is held by different managers, and a single (or a number below the protocol) data holder cannot restore the secret. For example, the process of implementing secret sharing on the original data s can include: first, select a security level parameter (system default or manual selection) and generate a corresponding finite field (for example, 2 256 ), then select random numbers in the finite field uniformly, and calculate s2 = s-s1, so that s1 and s2 are two slices of the original data s, which are distributed to two different managers.

[0056] Based on this, before this step, for the feature data or label data corresponding to a single sample in the part of the training data held by the P i party, the P i party can use the SS technology to split it into N secret sharing slices, keep one slice, and send the other N-1 slices to the other N-1 parties. Correspondingly, the P i party can also receive sample feature slices or sample label slices from the other N-1 parties. Further, the P i party can obtain the feature slices and label slices of each sample in the training sample set according to the slice it retains and the slices it receives from the other N-1 parties. For example, in the case of data horizontal division, multiple parties hold complete sample data of different business objects. At this time, the P i party can split the sample features and sample labels of the samples it holds, and receive feature slices and label slices of other samples from the other N-1 parties. Thus, the P i party holds the feature slices and label slices of all samples in the training set.

[0057] Further, in this step, for each batch in the current training, based on the first feature slice and the first label slice of the training sample of the batch, and the first parameter slice of the logistic regression model, a first loss slice is determined for restoring the training loss corresponding to the batch. It should be understood that each batch sample is usually multiple, and accordingly, the P iThe first feature slice can be composed of a plurality of feature slices corresponding to a plurality of samples in the batch, and the first label slice can be composed of a plurality of label slices corresponding to the plurality of samples in the batch. In addition, the first parameter slice is updated multiple times based on training samples in multiple batches. Specifically, when the batch is not the first batch in the current training, the first parameter slice is obtained based on the training and update of the last batch of samples; when the batch is the first batch in the current training, further, if the current training is the first period, the first parameter slice can be obtained by random initialization, and if the current training is not the first period, the first parameter slice can be obtained after the last period of training.

[0058] The above, P i The first feature slice, the first label slice and the first parameter slice held by the party are introduced for the convenience of mathematical description, or are respectively denoted as [x] i , [y] i and [θ] i . For determination of the first loss slice, P i based on [x] i , [y] i and [θ] i , and other N-1 parties use secure multi-party computation (MPC) technology to obtain collaborative calculation. In one calculation method, the calculation of the loss slice can be realized by designing confusion circuit, homomorphic operation, etc.

[0059] In another calculation method, Taylor expansion can be performed on the loss function of the LR model, so that only a small number of addition and multiplication operations are required to calculate the loss slice. Typically, the original loss calculation formula of the LR model is:

[0060]

[0061] In the above formula, S represents a sample subset formed by a batch of training samples; x i and y i represent the features and labels of the i-th sample in the sample subset S, and θ represents the model parameters.

[0062] First-order Taylor expansion of formula (1) can obtain the loss estimation formula:

[0063]

[0064] According to the above formula (2), P i party can obtain [x] i , [y] i and [θ] i, and other N-1 parties jointly perform secret sharing matrix multiplication (SMM) to obtain the first loss shard [L] of L(0) i In other words, each party calculates the loss shard by using SMM multiplication based on the feature shard, label shard and parameter shard held by each party.

[0065] The following is an example of two-party interaction to jointly calculate 0 T x i in formula (2) using SMM to calculate the loss shard. Figure 4 An interaction diagram showing two parties jointly performing SMM according to one embodiment is shown, where participating parties P A and P B , and P A holds the parameter shard [0 T ] A and the feature shard [x] A , and P B holds the parameter shard [0 T ] B and the feature shard [x] B .

[0066] As shown in Figure 4 , in step S41, P A and P B offline generate a Beaver multiplication triple, and P A has matrix shards [U]1, [V]1 and [W]1, and P B has matrix shards [U]2, [V]2 and [W]2, where [U]1, [U]2 have the same dimension as [0 T ] A and [0 T ] B , [V]1, [V]2 have the same dimension as [x] A and [x] B , and satisfy ([U]1 + [U]2) * ([V]1 + [V]2) = ([W]1 + [W]2). In step S42, P A calculates shards [D]1 and [E]1 and sends them to P B , P B calculates shards [D]2 and [E]2 and sends them to P A , and in step S43, P A restores D and E and calculates shard [Z]1, and P B restores D and E and calculates shard [Z]2. Where [Z]1 + [Z]2 = Z = 0T ·x. Thus, P A Fang and P B Based on their respective parameter slices and feature slices, Fang calculated θ. T The fragments [Z]1 and [Z]2 of x.

[0067] Furthermore, P A Fang calculated [Z]1, P B After calculating [Z]2, we can further calculate the values ​​for (1-2y). i )θ T x i Item fragmentation, specifically, based on Figure 4 The scheme shown takes [θ] T ] A and [x] A Replace them with (1-2[y]) respectively A ) and [Z]1, and [θ] T ] B and [x] B Replace them with (1-2[y]) respectively B ) and [Z]2. Therefore, P A Fang and P B Fang can jointly calculate (1-2y) i )θ T x i The partitioning of L(θ). Furthermore, the calculation of the log(2) term in equation (2) can be performed by either party; for the scalar multiplication term 1 / n, each party can calculate it locally. Thus, the two parties can collaboratively calculate the partitioning of L(θ). Specifically, P A The loss fragment [L] is obtained. A P B The loss fragment [L] is obtained. B .

[0068] Additionally, it is necessary to understand that P i Fang can be P A Fang or P B Fang, and, Figure 4 The proposed solution can be extended to scenarios involving more participants.

[0069] From the above, P i This allows us to obtain the first loss fragment [L] corresponding to each batch. i This yields m first loss slices corresponding to multiple batches (which can be denoted as m batches) in this training period. Used to reconstruct m training losses.

[0070] Then, in step S230, at P iIn the case that the P party is the designated party, the training loss of the current period is determined based on the plurality of training losses corresponding to the plurality of batches, and is used to evaluate the convergence state of the trained logistic regression model of the current period.

[0071] It should be understood that the reduction of the training loss and the evaluation of the convergence state of the model can be performed by a designated party from the plurality of parties, and the number of the designated parties can be one or more.

[0072] In one embodiment, in the P i In the case that the P party is the designated party, the training loss of the current period is determined based on the plurality of training losses corresponding to the plurality of batches, and is used to evaluate the convergence state of the trained logistic regression model of the current period.

[0073] Further, in a specific embodiment, a predetermined number of training losses can be randomly selected from the m training losses, and the average value is taken as the training loss of the current period. In another specific embodiment, the average value of the m training losses can be directly taken as the training loss of the current period.

[0074] In another embodiment, the P i The P party has a random seed pre-agreed with the other N-1 parties, and in this step, the P party randomly samples the m batches according to the random seed, and then jointly calculates the loss fractions corresponding to the randomly sampled batches with the other N-1 parties. Further, in the P i In the case that the P party is the designated party, the training loss of the current period is determined based on the plurality of training losses corresponding to the plurality of batches, and is used to evaluate the convergence state of the trained logistic regression model of the current period. i In the case that the P party is the designated party, the training loss of the current period is determined based on the plurality of training losses corresponding to the plurality of batches, and is used to evaluate the convergence state of the trained logistic regression model of the current period.

[0075] In this way, the training loss of the current period can be calculated. According to another aspect of the embodiment, after step S230, the method can further include determining the convergence state of the trained logistic regression model of the current period according to the training loss of the current period. In one embodiment, the convergence state can be determined only according to the training loss of the current period, for example, in the case that the training loss of the current period is less than a predetermined threshold, it is determined that the convergence has been achieved, otherwise it is determined that the convergence has not been achieved.

[0076] In another embodiment, the current trend of the training loss can be determined in combination with the past training loss, and then the convergence state can be more accurately determined according to the trend. In one embodiment, if the current training is not the first period, the change rate of the training loss can be determined according to the current training loss and the last training loss; further, in the case where the change rate is less than a preset threshold, the convergence state is determined to be converged, otherwise, the convergence state is determined to be not converged. Further, in a more specific embodiment, the change rate of the training loss can be calculated by the following formula:

[0077]

[0078] In the above formula, L j , L j-1 and L j-2 represent the training loss of the current period, the last period and the second last period respectively.

[0079] On the other hand, in the case where the convergence state is determined to be not converged, it can be further determined whether the convergence state is abnormal. In one specific embodiment, it is determined whether the current training loss is less than the last training loss, if not, it means that the model training is abnormal, and the convergence state is determined to be converged abnormal. In another specific embodiment, it is determined whether the change rate corresponding to the current training loss is less than the change rate corresponding to the last training loss, if not, it means that the model training is abnormal.

[0080] From the above, the convergence state of the LR model after the current training can be determined. Further, the convergence state can be used to assist the decision of the subsequent operation of the LR model. In one embodiment, in the case where the convergence state is converged, the training is stopped, and the LR model after the current training is determined as the trained LR model. That is, the current training is the last training of the LR model. In another embodiment, in the case where the convergence state is abnormal, the training is stopped, and the staff is prompted to adjust the hyperparameters used in the training, including the learning rate or the sample size per batch (Batch_size), etc., and then the training is continued after the adjustment. In still another embodiment, in the case where the convergence state is not converged, the training of the subsequent period is continued.

[0081] In summary, by using the method for training the LR model disclosed in the embodiments of the present specification, the training loss of each period is calculated under the condition of protecting the privacy input security, the convergence state of the LR model is timely and efficiently evaluated, which is used to guide the training process, so that the LR model with excellent performance is trained.

[0082] According to another aspect of the embodiments, the inventors also propose to evaluate the convergence state by calculating the norm of the model gradient. Figure 5A method flowchart for illustrating a multi-party joint training of a logistic regression model is shown according to another embodiment, wherein each of the multi-parties holds partial data forming a training sample set, and the method can be performed by any one of the multi-parties (or referred to as a first party, P i party), Figure 5 The steps involved in any one of the multiple training phases performed by the first party are shown in the following:

[0083] At step S510, a plurality of batches divided from the training sample set in the current phase are obtained; at step S520, for each batch of the plurality of batches, a first gradient slice is determined based on a first feature slice and a first label slice of the batch of training samples, and a first parameter slice of the logistic regression model, for updating the first parameter slice and restoring a corresponding training gradient; at step S530, in the case that the first party is a designated party, a training gradient norm of the current phase is determined based on a plurality of training gradients corresponding to the plurality of batches, for evaluating a convergence state of the logistic regression model after the training in the current phase.

[0084] The above steps are described in detail as follows:

[0085] Firstly, at step S510, a plurality of batches divided from the training sample set in the current phase are obtained. It should be noted that the description of step S510 can refer to the aforementioned description of step S210, which will not be repeated here.

[0086] At step S520, for each batch of the plurality of batches, a first gradient slice is determined based on a first feature slice [x] i and a first label slice [y] i of the batch of training samples, and a first parameter slice [θ] i of the logistic regression model. It should be noted that the description of [x] i , [y] i and [θ] i can refer to the aforementioned description in the embodiments, which will not be repeated here.

[0087] The determination of the first gradient slice [g ] can be obtained by P i party based on [x] i , [y] i and [θ] i , and other N-1 parties using MPC technology for collaborative calculation. In one calculation method, a garbled circuit, homomorphic operation, etc. can be designed to realize the calculation of the gradient slice. In another calculation method, Taylor expansion can be performed on the gradient calculation formula of the LR model, so that only addition and multiplication operations with small amount of calculation are needed to calculate the gradient slice. Typically, the gradient calculation formula of the LR model is:

[0088]

[0089] In the above formula, σ(·) represents a logic function.

[0090] When training the LR model jointly, it is very complex to directly use the above formula (4) to calculate the gradient, and therefore a linear approximation of the logic function is proposed, usually the Taylor expansion thereof, to simplify the gradient calculation, for example, using the first-order Taylor expansion of the logic function, see formula (5) below, to simplify the above formula (4) into the form of formula (6).

[0091]

[0092]

[0093] P i The P i th party can obtain the gradient shard i based on the [x] i , [y] and [θ] it holds, by jointly performing the SMM multiplication calculation formula (6) with the other N-1 parties, to obtain the gradient shard It should be noted that the specific calculation process can be designed according to the related description in the foregoing embodiments, and no further description is given.

[0094] In this way, the P i th party can obtain the gradient shard and update the parameter shard [θ] i using it, and the calculation formula can be:

[0095]

[0096] In the above formula, α represents the learning rate (or learning step size).

[0097] From the above, the P i th party can obtain the first gradient shard corresponding to each batch , and obtain m first gradient shads corresponding to m batches in the current training, denoted as m, for restoring the m training gradients.

[0098] After that, in step S530, in the case that the first party is a designated party, the training gradient norm of the current period is determined based on the plurality of training gradients corresponding to the plurality of batches, for evaluating the convergence state of the logistic regression model after the current training.

[0099] It should be understood that the restoration of the training gradient and the evaluation of the model convergence state can be performed by a designated party in the plurality of parties, and the number of the designated party can be one or more.

[0100] For the determination of the training gradient norm of the current period, the training gradient needs to be restored first, and then the gradient norm is determined. In an embodiment, in P i In the case that P is the designated party, for each batch of the m batches of the current period, other N-1 gradient slices corresponding to the batch are received from other N-1 parties, and the corresponding training gradient is restored. In this way, m training gradients corresponding to the m batches can be obtained.

[0101] Further, in a specific embodiment, a predetermined number of training gradients can be randomly selected from the m training gradients, and the norm (such as the two-norm) of each training gradient is calculated, and then the average of the gradient norms is calculated as the training gradient norm of the current period. In another specific embodiment, the norm of each training gradient in the m training gradients can be calculated, and then the average of the m gradient norms is calculated as the training gradient norm of the current period.

[0102] In another embodiment, P i The random seed previously agreed upon by other N-1 parties is stored in P i According to the random seed, the m batches are randomly sampled, and the gradient slices corresponding to the randomly sampled batches are calculated together with other N-1 parties. Further, in P i In the case that P is the designated party, other gradient slices for the sampled batches are received from other parties, thereby restoring the training gradient corresponding to the sampled batches, and then calculating the average of the norm of each training gradient as the training gradient norm of the current period.

[0103] In this way, the training gradient norm of the current period can be calculated. According to another embodiment, after step S530, the method can further include determining the convergence state of the logistic regression model after training of the current period according to the training gradient norm of the current period. In an embodiment, the convergence state can be determined only according to the training gradient norm of the current period, for example, in the case that the training gradient norm of the current period is less than a predetermined threshold, it is determined to have converged, otherwise it is determined to have not converged.

[0104] In another embodiment, the current change trend of the gradient norm can be determined in combination with the training gradient norm of the previous period, and then the convergence state is determined more accurately according to the change trend. In an embodiment, if the current training is not the first period, the change rate of the gradient norm can be determined according to the gradient norm of the current period and the gradient norm of the previous period; further, in the case that the change rate is less than a predetermined threshold, the convergence state is determined to have converged, otherwise it is determined to have not converged.

[0105] On the other hand, in the case where it is judged that the convergence state is not converged, it can be further judged whether the convergence state is abnormal. In one specific embodiment, it is judged whether the gradient norm of the current period is less than the gradient norm of the last period, and if not, it is indicated that the model training is abnormal, and the convergence state is determined to be convergence abnormal. In another specific embodiment, it is judged whether the change rate corresponding to the gradient norm of the current period is less than the change rate corresponding to the gradient norm of the last period, and if not, it is indicated that the model training is abnormal.

[0106] From the above, the convergence state of the LR model after the current training can be judged. Further, the subsequent operation on the LR model can be assisted by decision-making according to the convergence state. In one embodiment, in the case where the convergence state is converged, the training is stopped, and the LR model after the current training is determined as the trained LR model. In another embodiment, in the case where the convergence state is abnormal, the training is stopped, and the staff is prompted to adjust the hyperparameters used in the training, including the learning rate or the sample size per batch (Batch_size), etc., and after the adjustment, the training is continued. In still another embodiment, in the case where the convergence state is not converged, the training of the subsequent period is continued.

[0107] In summary, by using the method for jointly training the LR model disclosed in the embodiments of the present specification, the training gradient norm of each period is calculated under the condition of protecting the privacy input security of each party, the convergence state of the LR model is timely and efficiently evaluated, which is used to guide the training process, so as to train the LR model with excellent performance.

[0108] According to another aspect, the inventors also propose that the model convergence state is judged by calculating the validation loss of the LR model on the validation set after each period of training. Figure 6 Fig. 1 shows a flowchart of a method for jointly training a logistic regression model according to another embodiment, wherein each party holds part of the data forming the training sample set, and the method can be executed by any party (or first party, P i Fig. 1 shows a flowchart of a method for jointly training a logistic regression model according to another embodiment, wherein each party holds part of the data forming the training sample set, and the method can be executed by any party (or first party, P Figure 6 Fig. 1 shows a flowchart of a method for jointly training a logistic regression model according to another embodiment, wherein each party holds part of the data forming the training sample set, and the method can be executed by any party (or first party, P

[0109] Step S610, obtaining the batches of the training sample set in the current period; step S620, using the first training feature fragments and the first training label fragments of the multiple groups corresponding to the training samples of the multiple batches to perform multiple rounds of iterative updating on the first parameter fragments of the logistic regression model, to obtain the first parameter fragments updated in the current period; step S630, determining the first validation loss fragments based on the first validation feature fragments and the first validation label fragments corresponding to the validation sample set and the first parameter fragments updated in the current period, to restore the validation loss in the current period; and step S640, in the case that the first party is a designated party, evaluating the convergence state of the logistic regression model trained in the current period based on the validation loss in the current period.

[0110] The above steps are described as follows:

[0111] Firstly, in step S610, the batches of the training sample set in the current period are obtained. It should be noted that the description of step S610 can refer to the description of step S210, which will not be repeated here.

[0112] Step S620, using the first training feature fragments and the first training label fragments of the multiple groups corresponding to the training samples of the multiple batches to perform multiple rounds of iterative updating on the first parameter fragments of the logistic regression model, to obtain the first parameter fragments updated in the current period. Specifically, for each batch, the first training feature fragments and the first training label fragments of the training samples in the batch are used to calculate the first gradient fragments, and then the first gradient fragments are used to update the first parameter fragments of the logistic regression model. In this way, after multiple rounds of iterative updating corresponding to the multiple batches, the first parameter fragments updated in the current period can be obtained. It should be noted that the description of this step can also refer to the related description in the foregoing embodiments.

[0113] Step S630, determining the first validation loss fragments based on the first validation feature fragments and the first validation label fragments corresponding to the validation sample set and the first parameter fragments updated in the current period, to restore the validation loss in the current period.

[0114] In one embodiment, P i The party can stack the feature fragments corresponding to each validation sample in the validation sample set, so as to form a matrix by stacking, and take the matrix as the first validation feature fragments. Further, the party can stack the corresponding label fragments, so as to form a vector by stacking, and take the vector as the first validation label fragments. Further, the first validation loss fragments can be determined according to the first validation feature fragments and the first validation label fragments and the first parameter fragments updated in the current period.

[0115] In another embodiment, the complete validation sample set is pre-divided into multiple validation subsets, and sequentially corresponds to multiple training periods. Thus, the validation subset corresponding to the current training period can be determined from the multiple validation subsets, and the first validation feature slice and the first validation label slice of the validation subset are determined. Further, in one aspect, the first validation loss slice can be determined according to the determined first validation feature slice and the first validation label slice, and the first parameter slice updated in the current period. In another aspect, the first parameter slice updated in the current period can also be updated again according to the determined first validation feature slice and the first validation label slice, so as to fully utilize the calculation data of the validation slice.

[0116] It should be noted that the way of determining the first validation loss slice in this step can also refer to the introduction of determining the first loss slice in the foregoing embodiments.

[0117] From the above, the first validation loss slice can be obtained.

[0118] Step S640, in the case that the first party is the designated party, the convergence state of the logistic regression model after the current training is evaluated based on the current validation loss. It should be understood that the reduction of the validation loss and the evaluation of the model convergence state can be performed by the designated party in the multiple parties, and the number of the designated party can be one or more. In addition, for the above-mentioned current validation loss, other validation loss slices can be received from other parties in the multiple parties except the first party, and the current validation loss is reduced based on the first validation loss slice and the other validation loss slices.

[0119] In one embodiment, the convergence state can be directly determined according to the current validation loss, for example, in the case that the current validation loss is less than a preset threshold, it is determined that the convergence state is converged, otherwise it is determined that the convergence state is not converged.

[0120] In another embodiment, the current change trend of the validation loss can be determined in combination with the past validation loss, and the convergence state can be more accurately determined according to the change trend. In one embodiment, if the current validation is not the first period, the change rate of the validation loss can be determined according to the current validation loss and the last validation loss; further, in the case that the change rate is less than a preset threshold, the convergence state is determined to be converged, otherwise it is determined to be not converged.

[0121] On the other hand, in the case where it is judged that the convergence state is not converged, it can be further judged whether the convergence state is abnormal. In one specific embodiment, it is judged whether the current period validation loss is less than the last period validation loss, and if not, it is indicated that the model training is abnormal, and the convergence state is determined as convergence abnormality. In another specific embodiment, it is judged whether the change rate corresponding to the current period validation loss is less than the change rate corresponding to the last period validation loss, and if not, it is indicated that the model training is abnormal.

[0122] From the above, the convergence state of the LR model after the current period training can be judged. Further, the convergence state can be used to assist the decision of the subsequent operation of the LR model. In one embodiment, in the case where the convergence state is converged, the training is stopped, and the LR model after the current period training is determined as the trained LR model. That is, the current period training is the last period training of the LR model. In another embodiment, in the case where the convergence state is abnormal, the training is stopped, and the staff is prompted to adjust the hyperparameters used in the training, including the learning rate or the sample size per batch (Batch_size), etc., and the training is continued after the adjustment. In still another embodiment, in the case where the convergence state is not converged, the training of the subsequent period is continued.

[0123] In summary, by using the method for training the LR model disclosed in the embodiments of the present specification, the validation loss of each period training is calculated under the condition of protecting the privacy input security of each party, the convergence state of the LR model is timely and efficiently evaluated, which is used to guide the training process, so that the LR model with excellent performance is trained.

[0124] Corresponding to the above training method, the training device is also disclosed in the embodiments of the present specification. Figure 7 The structure schematic diagram of the device for training the logistic regression model according to one embodiment is shown, each party of the plurality of parties holds part of the data forming the training sample set, and the device is integrated in any first party of the plurality of parties; the device 700 performs any period of the plurality of periods of training involved in the training through the following units:

[0125] The batch acquisition unit 710 is configured to acquire a plurality of batches divided from the training sample set in the current period; the loss slice determination unit 720 is configured to determine, for each batch in the plurality of batches, a first loss slice based on the first feature slice and the first label slice of the batch training sample and the first parameter slice of the logistic regression model, the first loss slice being used to restore the training loss corresponding to the batch; the training loss determination unit 730 is configured to determine, in the case where the first party is a designated party, a current period training loss based on a plurality of training losses corresponding to the plurality of batches, the current period training loss being used to evaluate the convergence state of the logistic regression model after the current period training.

[0126] In an embodiment, the apparatus 700 further comprises a parameter shard updating unit 740, configured to determine a first gradient shard based on the first feature shard, the first label shard and the first parameter shard, and update the first parameter shard by using the first gradient shard.

[0127] In an embodiment, the training loss determining unit 730 is further configured to receive, from other parties in the plurality of parties except the first party, other loss shards corresponding to the each batch, and restore a corresponding training loss based on the first loss shard and the other loss shards corresponding to the each batch.

[0128] In an embodiment, the training loss determining unit 730 is specifically configured to randomly sample a predetermined number of training losses from the plurality of training losses, and take an average of the predetermined number of training losses as the current training loss.

[0129] In an embodiment, the apparatus 700 further comprises a convergence state evaluating unit 750 configured to determine the convergence state according to the current training loss, and a training process controlling unit 760 configured to stop training and determine a logistic regression model after current training as a trained logistic regression model in a case where the convergence state is converged.

[0130] In a specific embodiment, the current training is not the first period, and the convergence state evaluating unit 750 is specifically configured to determine a change rate of training loss according to the current training loss and a last training loss, and determine the convergence state as converged in a case where the change rate is less than a preset threshold.

[0131] In a specific embodiment, the training process controlling unit 760 is further configured to stop training and prompt a staff to adjust a hyperparameter used for training in a case where the convergence state is abnormal, the hyperparameter including a learning rate or a sample amount per batch.

[0132] Figure 8 An apparatus structure schematic diagram of training a logistic regression model by a plurality of parties according to another embodiment is shown, each of the plurality of parties holds part of data forming a training sample set, and the apparatus is integrated in a first party of the plurality of parties; the apparatus 800 performs any period of a plurality of training periods involved in the training by the following units:

[0133] The batch obtaining unit 810 is configured to obtain a plurality of batches for the training sample set in the current period. The gradient slice determining unit 820 is configured to, for each batch in the plurality of batches, determine a first gradient slice based on the first feature slice and the first label slice of the training samples in the batch and the first parameter slice of the logistic regression model, for updating the first parameter slice and restoring a corresponding training gradient. The gradient norm determining unit 830 is configured to, in the case that the first party is the first party, determine a training gradient norm in the current period based on a plurality of training gradients corresponding to the plurality of batches, for evaluating the convergence state of the logistic regression model after training in the current period.

[0134] In one embodiment, the gradient norm determining unit 830 is further configured to receive, from other parties in the plurality of parties except the first party, other gradient slices corresponding to the each batch; and restore the corresponding training gradient based on the first gradient slice and the other gradient slice corresponding to the each batch.

[0135] In one embodiment, the gradient norm determining unit 830 is specifically configured to: randomly sample a predetermined number of training gradients from the plurality of training gradients; calculate a norm corresponding to each training gradient in the predetermined number of training gradients; and obtain a mean value of the norms as the training gradient norm.

[0136] In one embodiment, the apparatus 800 further comprises: a convergence state evaluating unit 840 configured to determine the convergence state according to the training gradient norm; and a training process controlling unit 850 configured to, in the case that the convergence state is converged, stop training and determine the logistic regression model after training in the current period as a trained logistic regression model.

[0137] In one specific embodiment, the current training is not the first period; wherein the convergence state evaluating unit 840 is specifically configured to: determine a change rate of the training gradient norm according to the training gradient norm in the current period and the training gradient norm in the last period; and determine the convergence state as converged in the case that the change rate is less than a preset threshold.

[0138] In one specific embodiment, the training process controlling unit 850 is further configured to, in the case that the convergence state is abnormal, stop training and prompt a staff to adjust a hyperparameter used for training, the hyperparameter including a learning rate or a sample amount per batch.

[0139] Figure 9An apparatus structure diagram for training a logistic regression model by multiple parties according to another embodiment is shown, each of the multiple parties holding part of data forming a training sample set, the apparatus being integrated in a first party of the multiple parties; the apparatus 900 performs any one of multiple training periods involved in the training by the following units:

[0140] The batch obtaining unit 910 is configured to obtain multiple batches divided for the training sample set in the current period; the parameter slice determining unit 920 is configured to perform multiple rounds of iterative updating on the first parameter slice of the logistic regression model by using multiple sets of first training feature slices and first training label slices corresponding to the multiple batches, to obtain the first parameter slice updated in the current period; the loss slice determining unit 930 is configured to determine a first validation loss slice for restoring a validation loss in the current period, based on a first validation feature slice and a first validation label slice corresponding to a validation sample set, and the first parameter slice updated in the current period; the convergence state evaluating unit 940 is configured to, in the case that the first party is a designated party, evaluate a convergence state of the logistic regression model after training in the current period, based on the validation loss in the current period.

[0141] In one embodiment, the convergence state evaluating unit 940 is further configured to receive other validation loss slices from parties other than the first party among the multiple parties; and restore the validation loss in the current period based on the first validation loss slice and the other validation loss slices.

[0142] In one embodiment, the validation sample set is pre-divided into multiple validation subsets; the loss slice determining unit 930 is specifically configured to determine a validation subset corresponding to the training in the current period from the multiple validation subsets; and determine the first validation loss slice based on a first validation feature slice and a first validation label slice of the validation subset in the current period, and the first parameter slice updated in the current period; the apparatus further comprises a parameter slice updating unit 950 configured to update the first parameter slice updated in the current period according to the first validation feature slice and the first validation label slice.

[0143] In one embodiment, the training in the current period is not the first period; the convergence state evaluating unit 940 is specifically configured to determine a change rate of the validation loss according to the validation loss in the current period and a validation loss in a previous period; and determine that the convergence state is converged in the case that the change rate is less than a preset threshold.

[0144] In one embodiment, the apparatus 900 further comprises a training process control unit 960 configured to: in a case where the convergence state is converged, stop the training and determine the logic regression model after the current training as the trained logic regression model; or in a case where the convergence state is abnormal, stop the training and prompt a staff to adjust a hyper parameter used in the training, the hyper parameter including a learning rate or a sample amount per batch.

[0145] According to another aspect, embodiments further provide a computer readable storage medium having stored thereon a computer program which, when executed in a computer, cause the computer to perform the method described in conjunction with Figure 2 、 Figure 5 or Figure 6 .

[0146] According to yet another aspect, embodiments further provide a computing device comprising a memory and a processor, the memory having stored therein executable code that, when executed by the processor, implements the method described in conjunction with Figure 2 、 Figure 5 or Figure 6 .

[0147] Those skilled in the art should be aware that, in one or more examples described above, the functions described in the present application can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0148] The above detailed description sets forth the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above detailed description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.

Claims

1. A method for jointly training a logistic regression model by multiple parties, wherein each of the multiple parties holds a portion of the data forming a training sample set, and the method is applied to any first party among the multiple parties; The method involves multiple training phases, each of which includes: Obtain multiple batches from the training sample set in this period; For each of the multiple batches, a first loss slice is determined based on the first feature slice and the first label slice of the training samples in that batch, as well as the first parameter slice of the logistic regression model, to restore the training loss corresponding to that batch. When the first party is a designated party, the current training loss is determined based on the multiple training losses corresponding to the multiple batches, and is used to evaluate the convergence state of the logistic regression model after the current training. If the current training period is not the first period, the rate of change of training loss is determined based on the training loss in the current period and the training loss in the previous period. If the rate of change is less than a preset threshold, the convergence state is determined to be converged. Specifically, determining the current training loss based on the multiple training losses corresponding to the multiple batches includes: randomly selecting a predetermined number of training losses from the multiple training losses; and calculating the average value of the predetermined number of training losses as the current training loss.

2. The method according to claim 1, further comprising: Based on the first feature segmentation, the first label segmentation, and the first parameter segmentation, the first gradient segmentation is determined; Update the first parameter slice using the first gradient slice.

3. The method according to claim 1, wherein, Before determining the training loss for the current period based on the multiple training losses corresponding to the multiple batches, the method further includes: Receive other loss fragments corresponding to each batch from the parties other than the first party among the multiple parties; Based on the first loss segment and other loss segments corresponding to each batch, the corresponding training loss is restored.

4. The method according to claim 1, wherein, After determining the training loss for this period, the method further includes: If the convergence state is considered converged, training is stopped, and the logistic regression model trained in this period is determined as the trained logistic regression model.

5. The method according to claim 1, wherein, The method further includes: If the convergence state is abnormal, training is stopped, and staff are prompted to adjust the hyperparameters used for training, including the learning rate or the sample size per batch.

6. A method for jointly training a logistic regression model by multiple parties, wherein each of the multiple parties holds a portion of the data forming a training sample set, and the method is applied to any first party among the multiple parties; The method involves multiple training phases, each of which includes: Obtain multiple batches from the training sample set in this period; For each of the multiple batches, a first gradient slice is determined based on the first feature slice and the first label slice of the training samples in that batch, as well as the first parameter slice of the logistic regression model, and is used to update the first parameter slice and restore the corresponding training gradient. When the first party is a designated party, the training gradient norm of the current period is determined based on the multiple training gradients corresponding to the multiple batches, which is used to evaluate the convergence state of the logistic regression model after training in the current period. In the case that this training period is not the first period, the rate of change of the training gradient norm is determined based on the training gradient norm of this period and the training gradient norm of the previous period. If the rate of change is less than a preset threshold, the convergence state is determined to be converged. The step of determining the training gradient norm for the current period based on the multiple training gradients corresponding to the multiple batches includes: randomly selecting a predetermined number of training gradients from the multiple training gradients; calculating the norm of each training gradient in the predetermined number of training gradients; and taking the mean of the norms as the training gradient norm.

7. The method according to claim 6, wherein, Before determining the norm of the training gradient for the current period based on the multiple training gradients corresponding to the multiple batches, the method further includes: Receive other gradient fragments corresponding to each batch from the parties other than the first party among the multiple parties; Based on the first gradient slice and other gradient slices corresponding to each batch, the corresponding training gradients are restored.

8. The method according to claim 6, wherein, After determining the training gradient norm for this period, the method further includes: If the convergence state is considered converged, training is stopped, and the logistic regression model trained in this period is determined as the trained logistic regression model.

9. The method according to claim 8, wherein, The method further includes: If the convergence state is abnormal, training is stopped, and staff are prompted to adjust the hyperparameters used for training, including the learning rate or the sample size per batch.

10. A method for jointly training a logistic regression model by multiple parties, wherein each of the multiple parties holds a portion of the data forming a training sample set, and the method is applied to any first party among the multiple parties; The method involves multiple training phases, each of which includes: Obtain multiple batches from the training sample set in this period; Using multiple sets of first training feature slices and first training label slices corresponding to the multiple batches, the first parameter slices of the logistic regression model are iteratively updated in multiple rounds to obtain the updated first parameter slices in this period. Based on the first verification feature slice and the first verification label slice corresponding to the verification sample set, as well as the first parameter slice updated in this period, the first verification loss slice is determined. Receive other verification loss fragments from the parties other than the first party among the multiple parties; Based on the first verification loss fragment and other verification loss fragments, the original verification loss is restored; If the first party is a designated party, the convergence status of the logistic regression model trained in the current period is evaluated based on the current period validation loss. If the current training period is not the first period, the rate of change of the validation loss is determined based on the validation loss of the current period and the validation loss of the previous period. If the rate of change is less than a preset threshold, the convergence state is determined to be converged.

11. The method according to claim 10, wherein, The verification sample set is pre-divided into multiple verification subsets; wherein, based on the first verification feature slice and the first verification label slice corresponding to the verification sample set, and the first parameter slice updated in this period, the first verification loss slice is determined, including: From the plurality of validation subsets, determine the validation subset for the current training period that corresponds to the current training period; Based on the first verification feature slice and the first verification label slice of the current period's verification subset, and the first parameter slice updated in the current period, the first verification loss slice is determined. The method further includes updating the first parameter segment after the current update based on the first verification feature segment and the first verification label segment.

12. The method according to claim 10, wherein, After evaluating the convergence state of the logistic regression model trained in the current period based on the current period validation loss, the method further includes: If the convergence state is considered converged, training is stopped, and the logistic regression model trained in this period is determined as the well-trained logistic regression model; or, If the convergence state is abnormal, training is stopped, and staff are prompted to adjust the hyperparameters used for training, including the learning rate or the sample size per batch.

13. An apparatus for jointly training a logistic regression model by multiple parties, each of which holds a portion of data forming a training sample set, the apparatus being integrated into any first party among the multiple parties; the apparatus performing any one of the multiple training phases involved in the training through the following units: The batch acquisition unit is configured to acquire multiple batches divided for the training sample set in the current period; The loss segment determination unit is configured to determine a first loss segment for each of the plurality of batches, based on the first feature segment and the first label segment of the training samples in that batch, and the first parameter segment of the logistic regression model, so as to restore the training loss corresponding to that batch. The training loss determination unit is configured to, when the first party is a designated party, randomly select a predetermined number of training losses from the multiple training losses corresponding to the multiple batches and calculate the average value as the current period training loss, which is used to evaluate the convergence state of the logistic regression model after the current period training. The convergence state evaluation unit is configured to determine the rate of change of training loss based on the training loss of the current period and the training loss of the previous period when the current training period is not the first period; and to determine the convergence state as converged when the rate of change is less than a preset threshold.

14. An apparatus for jointly training a logistic regression model by multiple parties, each of which holds a portion of data forming a training sample set, the apparatus being integrated into any first party among the multiple parties; the apparatus performing any one of the multiple training phases involved in the training through the following units: The batch acquisition unit is configured to acquire multiple batches divided for the training sample set in the current period; The gradient partitioning determination unit is configured to determine a first gradient partition for each batch among the plurality of batches, based on the first feature partition and the first label partition of the training samples in that batch, and the first parameter partition of the logistic regression model, for updating the first parameter partition and restoring the corresponding training gradient. The gradient norm determination unit is configured to, when the first party is a designated party, randomly select a predetermined number of training gradients from the multiple training gradients corresponding to the multiple batches, calculate the norm corresponding to each training gradient, and take the mean of the norms as the training gradient norm of the current period, which is used to evaluate the convergence state of the logistic regression model after the current period of training. The convergence state evaluation unit is configured to determine the rate of change of the training gradient norm based on the training gradient norm of the current period and the training gradient norm of the previous period when the current training period is not the first period; and to determine the convergence state as converged when the rate of change is less than a preset threshold.

15. An apparatus for jointly training a logistic regression model by multiple parties, each of which holds partial data forming a training sample set, the apparatus being integrated into any first party among the multiple parties; the apparatus performing any one of the multiple training phases involved in the training through the following units: The batch acquisition unit is configured to acquire multiple batches divided for the training sample set in the current period; The parameter segmentation determination unit is configured to use multiple sets of first training feature segments and first training label segments corresponding to the multiple batches to perform multiple rounds of iterative updates on the first parameter segments of the logistic regression model to obtain the updated first parameter segments in the current period. The loss segment determination unit is configured to determine the first verification loss segment based on the first verification feature segment and the first verification label segment corresponding to the verification sample set, as well as the first parameter segment updated in this period, and to receive other verification loss segments from the other parties besides the first party among the multiple parties; Based on the first verification loss fragment and other verification loss fragments, the original verification loss is restored; The convergence state evaluation unit is configured to, when the first party is a designated party, evaluate the convergence state of the logistic regression model trained in the current period based on the current period validation loss; and, when the current period training is not the first period, determine the rate of change of the validation loss based on the current period validation loss and the previous period validation loss; and, if the rate of change is less than a preset threshold, determine the convergence state as converged.

16. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-12.

17. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-12.

Images