Method and device for joint updating model

By using the information filtering mechanism of the representation layer and mapping layer in federated learning, combined with the information bottleneck theory, the protection of data privacy and value in federated learning is achieved, the difficulties of data integration and privacy protection are solved, and data security and efficiency.

CN118484844BActive Publication Date: 2024-11-19ANT GROUP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410948445.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-19
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

In federated learning, how to achieve data integration and model updates between multiple training members without revealing data privacy, especially in the context of data silos and high costs, effectively protect data privacy and value.

Method used

Using the joint update model method, information filtering is performed through the characterization layer and the mapping layer, and using the information bottleneck theory to process data locally, maximize the mutual information between the characterization and the output, and minimize the mutual information between the characterization and the input, thereby Protect data privacy and value in federated learning.

Benefits of technology

It effectively reduces the amount of local sample data carried by model data uploaded to the service provider, improves the protection effect of data privacy and value, and ensures the security and efficiency of data in federated learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118484844B_ABST
    Figure CN118484844B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide a technical solution for jointly updating the model, in which the training members perform information filtering based on the information bottleneck theory at the representation layer and the mapping layer, and obtain the first representation by processing the sample feature data at the representation layer, and obtain the second representation by sampling the data through constructing the data distribution, and the sampled second representation is used to obtain the model output through the mapping layer processing, and the model loss is determined accordingly. At the same time, the model loss includes both the first loss determined by comparing the model output with the sample label data, and the second loss determined by comparing the data distribution satisfied by the second representation with the predetermined second distribution, so that in the process of minimizing the model loss, the mutual information between the second representation and the output can be maximized, and the mutual information between the second representation and the input can be minimized. In this way, the effectiveness of data privacy protection (or data value protection) can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the technical field of data security in joint computing, and in particular, to methods and devices for joint updating models. Background Art

[0002] With the rapid development of deep learning, artificial intelligence technology is showing its advantages in various industries. However, there are many difficulties in the reality of big data-driven artificial intelligence. For example, data silos are serious, utilization is low, and costs remain high. Single training members in some industries may also have problems with limited data or poor data quality. In addition, due to industry competition, privacy security, and complex management procedures, data integration between different departments of the same company may face huge resistance and high data integration costs.

[0003] Federated learning was proposed in this context. Federated learning is a framework based on distributed machine learning. The main idea is to build machine learning models based on data sets distributed on multiple devices while preventing data leakage. In this framework, clients (such as mobile devices) train models collaboratively under the coordination of the server, and the training data can be retained locally on the client, without the need to upload data to the data center like traditional machine learning methods. In the process of federated learning, how to protect the data privacy and data value of training members is a relatively important technical issue. Summary of the invention

[0004] One or more embodiments of this specification describe a method and device for jointly updating a model to solve one or more problems mentioned in the background technology.

[0005] According to a first aspect, a method for jointly updating a model is provided, wherein the model comprises a representation layer and a mapping layer, and the method is applicable to a single training member in horizontal federated learning, and the method comprises a plurality of synchronization cycles, wherein in a single synchronization cycle: feature data of a first sample is processed by the representation layer to obtain a first representation, wherein the first representation comprises an M-dimensional first vector and an M-dimensional second vector; an M-dimensional second representation is sampled, wherein a single element in the second representation is sampled from a single first distribution, and the first distribution is determined by a single element in the first vector and a single element in the second vector; the second representation is processed by the mapping layer to obtain a first output; and a model loss is determined to obtain synchronization by uploading model data to a service party according to the model loss, wherein the model loss comprises a first loss and a second loss, wherein the first loss is determined based on a comparison between the first output and the label data of the first sample, and the second loss is determined based on a comparison between a data distribution satisfied by the second representation and a predetermined second distribution.

[0006] In one embodiment, the first distribution is the following Gaussian distribution: a single element in the first vector is taken as a mean, and a mapping value of a single element in the second vector mapped to a non-negative data set is taken as a variance.

[0007] In a further embodiment, the variance is the square of a single element in the second vector, or the natural logarithm of the sum of a number whose base is e and whose exponent is 1.

[0008] In one embodiment, the second distribution is a standard normal distribution with a mean of 0 and a variance of 1.

[0009] In one embodiment, the model data is one of the model loss, the gradient of the pending parameter in the representation layer and the mapping layer, and the updated value of the pending parameter.

[0010] In one embodiment, the method further includes: receiving synchronization data of fusion feedback of model data provided by each training member from the service provider; and updating the pending parameters in the representation layer and the mapping layer using the synchronization data.

[0011] In one embodiment, the second loss is one of KL divergence, JS divergence, and cross entropy between the data distribution satisfied by the second representation and a predetermined second distribution.

[0012] According to a second aspect, a device for jointly updating a model is provided, the model comprising a representation layer and a mapping layer, the device is provided in a single training member in horizontal federated learning, the device comprises a representation unit, a sampling unit, a mapping unit, and a determination unit, and in a single synchronization cycle:

[0013] The representation unit is configured to process the feature data of the first sample through the representation layer to obtain a first representation, wherein the first representation includes an M-dimensional first vector and an M-dimensional second vector;

[0014] The sampling unit is configured to sample an M-dimensional second representation, wherein a single element in the second representation is sampled from a single first distribution, and the first distribution is determined by a single element in the first vector and a single element in the second vector;

[0015] The mapping unit is configured to process the second representation through the mapping layer to obtain a first output;

[0016] The determination unit is configured to determine the model loss so as to obtain model data upload to a service party for synchronization according to the model loss, wherein the model loss includes a first loss and a second loss, wherein the first loss is determined based on a comparison between the first output and the label data of the first sample, and the second loss is determined based on a comparison between a data distribution satisfied by the second representation and a predetermined second distribution.

[0017] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0018] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0019] Through the method and device provided by the embodiments of this specification, in the process of multiple training members jointly updating the model with the assistance of the service provider, the training members perform information filtering based on information bottlenecks at the representation layer and the mapping layer, and for the first representation obtained by processing the sample feature data at the representation layer, data sampling is performed by constructing a data distribution to obtain a second representation, and the sampled second representation is used to obtain the model output through processing at the mapping layer, and the model loss is determined thereby. At the same time, the model loss includes both the first loss determined by comparing the model output with the sample label data, and the second loss determined by comparing the data distribution satisfied by the second representation with the predetermined second distribution, so that in the process of minimizing the model loss, the mutual information between the second representation and the output can be maximized, and the mutual information between the second representation and the input can be minimized, so that the model data uploaded to the service provider carries as little data information in the local sample as possible, thereby improving the effectiveness of data privacy protection (or data value protection). BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1a A schematic diagram showing an implementation architecture of horizontal federated learning to which this specification applies;

[0022] Figure 1b A schematic diagram showing the model architecture of horizontal federated learning to which this specification applies;

[0023] Figure 2A schematic diagram of the model processing architecture on a single training member under the technical concept of this specification is shown;

[0024] Figure 3 A schematic diagram showing a process flow of a joint update model according to an embodiment of this specification;

[0025] Figure 4 A schematic block diagram showing an apparatus for jointly updating a model according to an embodiment of the present specification. DETAILED DESCRIPTION

[0026] The technical solution provided in this specification is described below in conjunction with the accompanying drawings.

[0027] Federated Learning, also known as federated machine learning, joint learning, and alliance learning, is a machine learning framework that can effectively help multiple institutions use data and conduct machine learning modeling while meeting user privacy protection, data security, and regulatory requirements.

[0028] Specifically, suppose that enterprise A and enterprise B each establish a task model, and a single task can be classification or prediction, and these tasks have been recognized by their respective users when obtaining data. However, due to incomplete data, such as enterprise A lacks label data, enterprise B lacks user feature data, or the data is insufficient and the sample size is insufficient to establish a good model, then the model at each end may not be established or the effect is not ideal. The problem that federated learning needs to solve is how to establish a high-quality model at each end of A and B. The training of this model uses the data of various enterprises such as A and B, and the data of each enterprise is not known to other parties, that is, to establish a shared model while minimizing the leakage of data privacy. This shared model is like a better model established by all parties by aggregating data together. In this way, the built model only serves its own goals in the area of ​​each party.

[0029] Each institution of federated learning can be called a training member (or data party, etc.). Each training member can hold different business data and can also participate in the joint training of the model through devices, computers, servers, etc. The business data here can be, for example, various data such as characters, pictures, voices, animations, videos, etc. Usually, the business data held by each training member is related, and the business parties corresponding to each training member can also be related. For example, among multiple business parties involved in financial business, business party 1 is a bank, which provides users with savings, loans and other services, and can hold data such as user income and expenditure flow, loan amount, deposit amount, etc. Business party 2 is a shopping website, which holds data such as user shopping habits, payment habits, payment accounts, etc. For another example, among multiple business parties involved in medical business, each business party can be a hospital, physical examination institution, etc. For example, business party 1 is hospital A, corresponding to user symptoms, diagnosis results, treatment plans, treatment results, etc. diagnosis and treatment records as local business data, business party 2 can be physical examination institution B, corresponding to user symptoms, physical examination conclusions, etc. physical examination record data, and so on. A single training member of federated learning can hold the business data of one business party or multiple business parties.

[0030] Under the federated learning architecture, two or more data parties can jointly train the model. The model here can be various models used to process business data to obtain corresponding business processing results, which can also be called business models. For example, business data can be user financial-related data, and the business processing result obtained is the user's financial credit evaluation result. For another example, business data can be the user's customer service conversation data, and the business processing result obtained is the recommendation result of the customer service answer, and so on. Each training member can use the local business model to perform local data processing on the local business data. The purpose of federated learning is to train a model that can better process these business data, so the federated learning model can also be called a business model.

[0031] Federated learning is divided into horizontal federated learning and vertical federated learning. Under the horizontal federated learning framework, the sample features of different training members in the sample sets have a high overlap, but the sample sources are different. For example, multiple sample sets correspond to customers of different banks. Generally, the data features managed by different banks may be similar, but the customers may be different. In this way, when each training member holds the business data of different banks, the model can be trained in the form of horizontal federated learning. In vertical federated learning, the IDs of different data sets (such as phone numbers, etc.) have a high overlap (for example, the records of phone numbers, etc. are consistent), but the feature data are different. For example, a bank and a hospital with the same user group (such as residents of a small county town) have a large number of people in the samples of the bank and the hospital with a high overlap, but the features are different. The bank data may correspond to feature information such as deposits and loans, and the hospital data may correspond to feature information such as physiological indicators, health status, and medical records. Here, the training members who hold the business data of the bank and the hospital respectively can use the vertical federated learning method to jointly train the model.

[0032] The technical solution provided in this specification proposes improvements to the model training method under the horizontal federated learning architecture. Figure 1a Figure 2 shows a system implementation architecture of horizontal federated learning. Figure 1a As shown in the figure, each business party, such as business party 1 and business party 2, holds local training data (including sample feature data and sample label data) and can participate in federated learning as a training member. Each training member trains the business model locally, and then uploads the updated model data (such as the pending model parameters or the gradient data for updating the model parameters) to the service party. The service party is used to provide auxiliary computing in the federated learning process, such as synchronizing the data uploaded by each training member, and then sending the synchronized model data to each business party.

[0033] Figure 1b A schematic diagram of the model architecture in the horizontal federated learning scenario is also shown. It can be understood that in the federated learning architecture, the training members (for example, Figure 1a The model held by the business party in the training is called a local model, and the model held by the service party is called a global model. In the horizontal federated learning architecture, the local models held by each training member are consistent with the global model held by the service party, for example, they have the same number of model layers and parameters. The global model does not necessarily have to exist, but can be represented only by a global parameter set. Each training member can train a local model locally using its own training data and update the pending parameters. In each synchronization cycle, the training member can update the locally updated pending parameters, or the model data such as the gradient data used to update the pending parameters ( Figure 1b The server sends the local parameter set to the server, which then trains the model data of each training member ( Figure 1b The global parameter set is synchronized and then fed back to each training member.

[0034] Given that the model data uploaded by the training members may contain knowledge beyond the current training task, the data value of the training members may be leaked. In order to avoid the leakage of the data value of the training members, the training members may perform relevant data processing locally before uploading the model data to the service provider, such as encryption and sparsification under conventional technical means. Considering that the models in federated learning are usually classification models or prediction models, the models are often divided into a representation layer and a mapping layer, wherein the representation layer is used to process the sample feature data to mine high-order features, thereby forming a deep representation of the sample, and the mapping layer classifies or predicts the sample based on the deep representation. Under the technical concept of this specification, a data value protection method based on information bottleneck is proposed.

[0035] The so-called information bottleneck means that when faced with a task, try to use the least amount of information to complete it. Usually, a task can be divided into two parts, one part is used to process the input data, such as through a representation layer that encodes the features, and the other part maps the processing results of the input data to output data, such as through an activation function that maps the processing results to a mapping layer of probability values. Imagine adding a "bottleneck" θ to the processing results of the input data, similar to an hourglass. The amount of information entering may be a lot, but the amount of information going out is limited by θ. In other words, the role of this bottleneck is: not allowing more information to flow through than θ. After the information passes the information bottleneck, it still needs to complete the task it is supposed to complete (such as classification, regression, etc.), so the most important information needs to pass through the bottleneck.

[0036] According to the information bottleneck theory, during the information transmission process, the bottleneck part of the system can limit the flow and processing of information. In machine learning, the information bottleneck theory emphasizes improving the generalization ability of the learning system by reducing the redundancy between input and output information. By introducing compressed representation, the goal of the information bottleneck is to reduce unnecessary complexity while retaining key information. Specifically for federated learning tasks, consider when training members train models locally, such as Figure 2 As shown, an information bottleneck layer is inserted between the representation layer and the mapping layer of the model. In this way, on the one hand, the mutual information between the representation and the output can be maximized by comparing the model output with the sample label. On the other hand, a new representation can be sampled from the representation output of the representation layer for the mapping layer to determine the model output. By making the sampled representation approach a predetermined distribution, the mutual information between the representation and the input is minimized, thereby striving to minimize information out of the domain, thereby protecting the data value of the training members.

[0037] Combine the following Figure 3 The illustrated embodiments describe the technical concept of the present specification in detail.

[0038] Figure 3 The process of the joint update model according to an embodiment of this specification is shown. The execution subject of the process can be any computer, device, or server with a certain computing power. Specifically, for example, a single training member in horizontal federated learning. Among them, the implementation architecture of the joint update model is as follows Figure 1a As shown, each training member can update the local model locally and upload the model data to the service provider for synchronization according to the synchronization cycle. The model data here can be model loss, pending parameters, gradient data of pending parameters, etc. Among them, the model loss can measure the degree of deviation between the result obtained by the model performing the corresponding task on the training sample and the expected result contained in the sample. The pending parameters can be the parameters adjusted during the model training process to minimize the model loss. The gradient data of the pending parameters is the data used to describe the adjustment direction in the adjustment method based on the gradient descent method, Newton method, etc.

[0039] For horizontal federated learning, the model on a single training member may include at least a representation layer and a mapping layer. The representation layer is used to process the input data to obtain sample representations, and the mapping layer can map the sample representations to probability values ​​on the predetermined tasks, thereby determining the model output. In the process of each training member and the service provider jointly updating the model, multiple synchronization cycles may be included. In a single synchronization cycle, each training member uploads local model data to the service provider once. A single synchronization cycle may correspond to an update cycle for pending parameters in one or more models. In a single update cycle, a single training member determines the local model loss through a batch of training samples, and adjusts the pending parameters based on the local model data or the synchronized model data.

[0040] like Figure 3 As shown, the process of jointly updating the model performed by a single training member may include the following steps: Step 301, processing the feature data of the first sample through the representation layer to obtain a first representation, the first representation including an M-dimensional first vector and an M-dimensional second vector; Step 302, sampling the first vector and the second vector to obtain an M-dimensional second representation, wherein a single element in the second representation is sampled in a Gaussian distribution determined by a single mean and a single variance, the single mean is determined by the corresponding element in the first vector, and the single variance is determined by the corresponding element in the second vector; Step 303, processing the second representation tensor through the mapping layer to obtain a first output; Step 304, determining the model loss, and uploading the model data of the model obtained according to the model loss to the service party for data synchronization, the model loss including a first loss and a second loss, the first loss is determined based on the comparison between the first output and the first label of the first sample, and the second loss is determined based on the comparison between the data distribution satisfied by the second representation and the standard normal distribution.

[0041] First, in step 301, feature data of a first sample is processed by a representation layer to obtain a first representation, where the first representation includes an M-dimensional first vector and an M-dimensional second vector.

[0042] Here, the first sample can be any training sample in the current batch of training samples. The first sample can correspond to feature data (such as X) and label data (such as Y). For example, a financial user sample can correspond to financial information such as consumption records, shopping preferences, historical borrowing times, borrowing cycles, and historical compliance times. Corresponding feature values ​​can be extracted from this information to form feature data to describe user financial information. On the other hand, the label data can be determined based on whether the single borrowing behavior corresponding to the feature data is in compliance, such as 1 for compliance and 0 for non-compliance.

[0043] The representation layer may be a neural network for encoding and representing feature data. The neural network may be formed by at least one of a fully connected network, an attention network, etc. The processing result of the feature data by the representation layer may be recorded as a representation tensor. The representation tensor for the first sample may be recorded as a first representation, for example.

[0044] Under the technical concept of this specification, the numerical values ​​of each dimension of the first representation can be used as sampling parameters for sampling a new representation, for example, to determine the distribution satisfied by the sampled data (such as recorded as the first distribution). In one embodiment, in order to sample a new representation of M dimensions, a distribution can be determined by the numerical values ​​of two dimensions of the first representation. For example, the first representation can be a 2M-dimensional vector, or it can be a vector with two dimensions of M, which are uniformly recorded as two M-dimensional vectors in this specification, such as the first vector M1 and the second vector M2, respectively. Among them, M is a natural number greater than 1. In other words, the elements between the first vector and the second vector can correspond one to one. The first representation can describe the feature data in the first sample from a higher-order dimension. The first representation carries more information about the sample feature data.

[0045] Then, through step 302, a second representation of M dimensions is obtained by sampling the first vector and the second vector.

[0046] In order to filter out the effective information in the first representation, an M-dimensional second representation (such as V) can be obtained based on the sampling of the first vector and the second vector. Specifically, a single element in the second representation can be sampled from a first distribution determined by a single element in the first vector and a single element in the second vector. For example, the i-th (i is a natural number between 1 and M) element in the second representation (such as V i ) can be represented by the i-th element in the first vector (denoted as M 1i ) and the i-th element in the second vector (denoted as M 2i ) is sampled from the first distribution determined by .

[0047] Here, the first distribution may be a predetermined distribution, which may be determined by two parameters, such as a Gaussian distribution, a Laplace distribution, etc. A person skilled in the art may understand that the two parameters determining the Gaussian distribution are the mean and the variance, and the two parameters determining the Laplace distribution are the location parameter and the scale parameter. The two parameters in the first distribution may correspond to a single element in the first vector and a single element in the second vector, respectively. Taking the first distribution as a Gaussian distribution as an example, for the i-th element V in the second representation i For example, the corresponding two parameters are respectively denoted as mean μ i , and the variance σ i 2 Among them, the mean μ i For example, it can be the i-th element M in the first vector 1i Multiples of, such as μ i =α·M 1i , usually α is 1, but other values ​​are possible. Variance σ i 2 is a positive number, and the i-th element in the second vector may be positive or negative, so the variance σ i 2 It can be the i-th element M in the second vector 2i The positive number converted into a predetermined number, such as σ i 2 =M 2i 2 , σ i 2 =ln(1+ M 2i 2 ) and so on. Taking the first distribution as Laplace distribution as an example, for the i-th element V in the second representation i For example, the corresponding two parameters are the position parameter μ i , and the scale parameter λ i Among them, the location parameter μ i It can be positive, negative or 0, so the position parameter μ i The construction method of is similar to the mean in the Gaussian distribution mentioned above, such as μ i =α·M 1i , the scale parameter λ i It is usually a number greater than 0, and its construction method is similar to the variance in the Gaussian distribution mentioned above, such as λ i =M 2i 2 , i =ln(1+ M 2i 2 ) and so on. In practice, the first distribution may also be other reasonable distributions, which will not be described in detail here.

[0048] In this way, sampling is performed on each dimension of the second representation to obtain an M-dimensional second representation. The second representation can replace the first representation for data processing of the first sample.

[0049] Next, according to step 303, the second representation is processed by the mapping layer to obtain a first output.

[0050] It can be understood that the mapping layer can map the representation to the probability of the classification or candidate items through the activation function, so that the output layer outputs the mapped probability or the category determined based on the mapped probability. For example, when using financial data to predict the financial risk of users, the mapping layer can map the representation to a probability value on at least one of the classification categories of "risky" and "risk-free", and the output layer can determine the output result according to the size of the probability value. If the mapping layer obtains the probability values ​​of 0.8 and 0.2 for the two classification categories of "risky" and "risk-free" respectively, the output layer outputs a risky prediction result (such as outputting 0 for risky). Here, the data output by the output layer after processing the second representation through the mapping layer can be recorded as the first output (such as marked as ).

[0051] Further, using step 304, the model loss is determined, and the model data of the model obtained according to the model loss is uploaded to the service provider for data synchronization.

[0052] It can be understood that under the information bottleneck theory, the optimization goal of a single training member for the model may include: maximizing the mutual information between the second representation and the output, while minimizing the mutual information between the second representation and the input.

[0053] In probability theory and information theory, mutual information (MI) is a concept that measures the correlation between two sets of events. The mutual information or transfer information of two random variables is a measure of the mutual dependence between the variables. Unlike the correlation coefficient, mutual information is not limited to real-valued random variables. It is more general and can describe the similarity between the joint distribution p(X, Y) and the product of the decomposed marginal distribution p(X)p(Y).

[0054] In order to maximize the mutual information between the second representation and the output, the first output , with the label data of the first sample By comparison, we can get the first loss in the model loss, which is recorded as In practice, samples are usually processed in batches, and the output of a single batch is recorded as a tensor , Can be part of, such as a vector A dimension in , or a matrix A row in, etc. The label data Y of a batch of samples can be With the same dimensions, y can be Therefore, the first loss It can be determined by comparing the similarity of the tensors, for example, the cross entropy ( ), Euclidean distance, cosine similarity, etc. Taking cross entropy as an example, if recorded as .

[0055] On the other hand, in order to minimize the mutual information between the second representation and the input, it can be assumed that the distribution of the M dimensions of the second representation is: P(V|X). Let the distribution P(V|X) approach a predetermined second distribution Q(Z), thereby determining the second loss in the model loss (such as denoted as ). Here, the distribution P(V|X) satisfied by the second characterization can be determined according to the second distribution Q(Z). For example, when the second distribution is a Gaussian distribution, the distribution P(V|X) can be fitted as a Gaussian distribution. When the second distribution is a Laplace distribution, the distribution P(V|X) can be fitted as a Laplace distribution. Optionally, for ease of description and calculation processing, the second distribution can select a standard distribution, such as a Gaussian distribution with a mean of 0 and a variance of 1 for Q(Z), or a Laplace distribution with a location parameter of 1 and a size parameter of 1.

[0056] The comparison between the distribution P(V|X) and the second distribution Q(Z) can be measured by KL divergence (relative entropy), JS divergence, cross entropy and other indicators. Taking KL divergence as an example, it is recorded as = .

[0057] The model loss may be determined by the first loss and the second loss, for example, the sum or weighted sum of the first loss and the second loss. As a specific example, the model loss may be expressed as: .in, It is a weight coefficient or a balance coefficient, which is used to adjust the proportion of the first loss and the second loss.

[0058] In this way, while minimizing the model loss, the mutual information between the second representation and the model output can be maximized, and the mutual information between the second representation and the input data can be minimized.

[0059] Model data can be obtained based on model loss. The model data here can be data used by the service provider for data synchronization, such as model loss, pending parameter update values ​​of the representation layer and mapping layer, gradients of pending parameters of the representation layer and mapping layer, etc. Among them, the gradients of the pending parameters of the representation layer and mapping layer are the derivatives obtained by differentiating the pending parameters through the model loss, which are used to describe the update direction of the pending parameters. The update values ​​of the pending parameters are obtained by updating the corresponding gradients according to the gradient update methods of the parameters such as the gradient descent method and the Newton method, which will not be repeated here.

[0060] It is worth noting that the synchronization period of the model data and the update period of the pending parameters may be consistent or inconsistent. The synchronization period of the model data may be determined by a synchronization condition. The synchronization condition is, for example, at least one of the following: a predetermined number (such as 3, 10, etc.) of update periods have passed since the start time or the last synchronization, and a predetermined time interval (such as 1 minute, etc.) has arrived.

[0061] A single training member can provide model data to the service provider if the synchronization conditions are met, and the service provider will perform data synchronization. The process of data synchronization by the service provider can be a process in which the service provider fuses the model data provided by each training member in at least one of averaging, median, maximum, minimum, and weighted average. The service provider is usually a trusted third party. After synchronizing the model data, it can directly feed back to the training members, or it can feed back to the training members after processing. This specification does not limit this. The processing here can be processing the synchronized model data itself in a manner such as sparseness and encryption, or it can be processing after subsequent calculations. For example, when the model data is a model loss, the service provider can further calculate the gradient of the pending parameters, or the updated value of the pending parameters, etc. and feed back to the training members.

[0062] It is worth noting that the words "first", "second", etc. that appear in the description of this specification are for the convenience of description and are used to distinguish related concepts in the technical solution. They do not indicate the order and do not constitute a substantial limitation on the related concepts.

[0063] Reviewing the above process, the technical concept of the joint update model provided by the embodiment of this specification is that the training members perform information filtering based on information bottlenecks at the representation layer and the mapping layer, and obtain the first representation by processing the sample feature data at the representation layer, and obtain the second representation by data sampling through constructing the data distribution. The sampled second representation is used to obtain the model output through the mapping layer processing, and the model loss is determined accordingly. At the same time, the model loss includes both the first loss determined by comparing the model output with the sample label data, and the second loss determined by comparing the data distribution satisfied by the second representation with the predetermined second distribution. In the process of minimizing the model loss, the mutual information between the second representation and the output can be maximized, and the mutual information between the second representation and the input can be minimized, so that the model data uploaded to the service provider carries as little data information in the local sample as possible, thereby improving the effectiveness of data privacy protection (or data value protection).

[0064] According to another embodiment, a device for jointly updating a model is also provided. The device for jointly updating a model may be provided in a single training member in horizontal federated learning, and the single training member may correspond to a computer, device or server with a certain computing power. Figure 4 As shown, the device 400 for jointly updating a model may include a characterization unit 401 , a sampling unit 402 , a mapping unit 403 , and a determination unit 404 .

[0065] This single training member can perform multiple synchronization cycles with other training members. In a single synchronization cycle:

[0066] A representation unit 401 is configured to process feature data of a first sample through a representation layer to obtain a first representation, where the first representation includes an M-dimensional first vector and an M-dimensional second vector;

[0067] A sampling unit 402 is configured to sample an M-dimensional second representation, wherein a single element in the second representation is sampled from a single first distribution, the first distribution being determined by a single element in the first vector and a single element in the second vector;

[0068] A mapping unit 403 is configured to process the second representation through a mapping layer to obtain a first output;

[0069] The determination unit 404 is configured to determine the model loss so as to obtain the model data according to the model loss and upload it to the service party for synchronization. The model loss includes a first loss and a second loss. The first loss is determined based on the comparison between the first output and the label data of the first sample, and the second loss is determined based on the comparison between the data distribution satisfied by the second representation and the predetermined second distribution.

[0070] According to one embodiment, the first distribution is the following Gaussian distribution: the single element in the first vector is the mean, and the single element in the second vector is mapped to the non-negative data set as the variance. For example, the variance corresponding to the single element can be the square of the single element in the second vector, or the natural logarithm of the sum of the number with e as the base and the single element in the second vector as the exponent and 1 .

[0071] According to one embodiment, the second distribution may be a standard normal distribution with a mean of 0 and a variance of 1.

[0072] According to one embodiment, the model data is one of the model loss, the gradient of the pending parameters in the representation layer and the mapping layer, and the updated value of the pending parameters.

[0073] According to one embodiment, the apparatus 400 may further include an updating unit configured to: receive synchronization data of fusion feedback of the service provider based on the model data provided by each training member; and update the pending parameters in the representation layer and the mapping layer using the synchronization data.

[0074] According to one embodiment, the second loss is one of KL divergence, JS divergence, and cross entropy between the data distribution satisfied by the second representation and a predetermined second distribution.

[0075] It is worth mentioning that Figure 4 The device 400 shown is Figure 3 The method flow described corresponds to Figure 3 The corresponding description in the illustrated method embodiment is also applicable to the device 400 and will not be repeated here.

[0076] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 3 The method described by et al.

[0077] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 3 The method described by et al.

[0078] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of this specification can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0079] The specific implementation methods described above further explain in detail the purpose, technical solutions and beneficial effects of the technical concept of this specification. It should be understood that the above description is only the specific implementation method of the technical concept of this specification, and is not used to limit the protection scope of the technical concept of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of this specification should be included in the protection scope of the technical concept of this specification.

Claims

1. A method for jointly updating a model, the model comprising a representation layer and a mapping layer, an information bottleneck layer being arranged between the representation layer and the mapping layer, the method being applicable to a single training member in horizontal federated learning, the method comprising a plurality of synchronization cycles, in a single synchronization cycle: Processing the feature data of the first sample through the representation layer to obtain a first representation, wherein the first representation includes an M-dimensional first vector and an M-dimensional second vector; Sample the second representation of M dimensions, where The single element in the second representation is sampled from a single first distribution, the first distribution being one of the following: a Gaussian distribution with a single element in the first vector as the mean and a single element in the second vector mapped to a non-negative data set as the mapping value determined by the variance; a Laplace distribution with a single element in the first vector as the location parameter and a single element in the second vector as the scale parameter; Processing the second representation through the mapping layer to obtain a first output; Determine the model loss to obtain model data uploaded to the service party for synchronization according to the model loss, wherein the model loss includes a first loss and a second loss, wherein the first loss is determined based on a comparison between the first output and the label data of the first sample, and the second loss is determined based on a comparison between a data distribution satisfied by the second representation and a predetermined second distribution.

2. The method of claim 1, wherein: The variance is the square of a single element in the second vector, or the natural logarithm of the sum of a number with e as the base and 1 as the exponent of the single element in the second vector.

3. The method of claim 1, wherein: The second distribution is a standard normal distribution with a mean of 0 and a variance of 1.

4. The method of claim 1, wherein: The model data is one of the model loss, the gradient of the pending parameter in the representation layer and the mapping layer, and the updated value of the pending parameter.

5. The method of claim 1, wherein: The method further comprises: Receive synchronization data from the service provider based on the fusion feedback of model data provided by each training member; The synchronization data is used to update the pending parameters in the representation layer and the mapping layer.

6. The method of claim 1, wherein: The second loss is one of KL divergence, JS divergence, and cross entropy between the data distribution satisfied by the second representation and a predetermined second distribution.

7. A device for jointly updating a model, the model comprising a representation layer and a mapping layer, an information bottleneck layer being arranged between the representation layer and the mapping layer, the device being arranged in a single training member in horizontal federated learning, the device comprising a representation unit, a sampling unit, a mapping unit, and a determination unit, in a single synchronization cycle: The representation unit is configured to process the feature data of the first sample through the representation layer to obtain a first representation, wherein the first representation includes an M-dimensional first vector and an M-dimensional second vector; The sampling unit is configured to sample an M-dimensional second representation, wherein a single element in the second representation is sampled from a single first distribution, and the first distribution is one of the following: a Gaussian distribution determined by a single element in the first vector as a mean and a mapping value of a single element in the second vector mapped to a non-negative data set as a variance; a Laplace distribution with a single element in the first vector as a location parameter and a single element in the second vector as a scale parameter; The mapping unit is configured to process the second representation through the mapping layer to obtain a first output; The determination unit is configured to determine the model loss so as to obtain model data upload to a service party for synchronization according to the model loss, wherein the model loss includes a first loss and a second loss, wherein the first loss is determined based on a comparison between the first output and the label data of the first sample, and the second loss is determined based on a comparison between a data distribution satisfied by the second representation and a predetermined second distribution.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

9. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 6 is implemented.