Method and apparatus for training a model

Abstract

This specification provides a method and apparatus for jointly updating a model. When training members provide intermediate tensors to the service provider, the intermediate tensors are first sparsified. Then, the effective elements after sparsification are quantized to reduce the number of bytes, thereby providing the sparsed data of the intermediate tensors to the service provider with a smaller amount of data. The service provider, upon receiving the sparsed data, first performs dequantization to recover the sparse tensor corresponding to the intermediate tensor, and then performs further processing. Afterwards, the service provider feeds back the processed tensor corresponding to the intermediate tensor of each training member based on the subsequent processing, allowing each training member to update its local model. This method further quantizes the sparsified intermediate tensors, significantly reducing the number of bytes in the values, thereby reducing the amount of communication data, shortening communication time, and improving model training efficiency.

Description

Technical Field

[0001] This specification relates to the field of secure computing technology, and more particularly to methods and apparatus for jointly training models. Background Technology

[0002] With the rapid development of deep learning, artificial intelligence technology is demonstrating its advantages in almost every industry. However, big data-driven AI faces many challenges in reality. For example, data silos are severe, resulting in low utilization and persistently high costs. Individual training members in some industries may also have limited or poor-quality data. Furthermore, due to industry competition, privacy concerns, and complex management procedures, data integration between different departments within the same company can face significant obstacles and high costs.

[0003] Federated learning was proposed in this context. Federated learning is a framework based on distributed machine learning, whose main idea is to build machine learning models based on datasets distributed across multiple devices while preventing data leakage. In this framework, clients (e.g., mobile devices) collaboratively train the model under the coordination of a central server, while the training data can be stored locally on the client, eliminating the need to upload data to a data center as in traditional machine learning methods. During federated learning, the amount of data transmitted is usually proportional to the sample size; the larger the data volume, the greater the communication volume. For large-scale joint learning, excessive communication volume can lead to a longer overall training time. Summary of the Invention

[0004] This specification describes one or more embodiments of a method and apparatus for jointly training a model to solve one or more problems mentioned in the background art.

[0005] According to a first aspect, a method for jointly training a model is provided, in which a service provider and multiple training members jointly update the model using their local privacy data, wherein the multiple training members include a first member, which holds first feature data for training samples, and the model has a first local model corresponding to the first member and a global model corresponding to the service provider; the method is executed by the first member, and in the current model update cycle, the method includes: processing the first feature data of several training samples in the current batch based on the first local model to obtain a first intermediate tensor; sparsifying the first intermediate tensor to obtain a first sparse tensor described by a first numerical type; quantizing the elements retained in the first sparse tensor to obtain first sparse data, wherein the first sparse data describes the corresponding elements by a second numerical type, and the number of bytes corresponding to the first numerical type is greater than the number of bytes corresponding to the second numerical type; providing the first sparse data described by the second numerical type to the service provider, so that the service provider can process the inverse quantization result of the first sparse data accordingly and feed back a first processed tensor corresponding to the first intermediate tensor; and updating the first local model based on the first processed tensor.

[0006] In one embodiment, in longitudinal federated learning, the service provider is a training member or a trusted third party holding labeled data, the global model is sequentially connected to the first local model, and the first intermediate tensor is the processing result of the first local model on the first feature data of the plurality of training samples.

[0007] In one embodiment, the first processing tensor is: the first intermediate gradient of the model loss with respect to the first intermediate tensor, wherein the first intermediate gradient is obtained by the service provider based on the dequantization of the first sparse data, the second intermediate tensor is processed by the global model to obtain the fusion result of the second intermediate tensor and other intermediate tensors fed back by other training members, and the corresponding prediction result is obtained. The model loss is determined by comparing the prediction result with the label data, thereby determining the first intermediate gradient through gradient backpropagation; the step of updating the first local model based on the first processing tensor further includes: using the first intermediate gradient to determine the gradients corresponding to each undetermined parameter in the first local model; updating each undetermined parameter in the first local model according to each gradient, thereby updating the first local model.

[0008] In one embodiment, in horizontal federated learning, the service provider is a trusted third party, the global model has the same structure as the first local model, and the first intermediate tensor is a tensor used to describe the parameters to be synchronized. The parameters to be synchronized are one of the following: the change of the parameters to be determined, the gradient of the parameters to be determined, and the updated value of the parameters to be determined.

[0009] In one embodiment, the first processing tensor is a fusion tensor obtained by the service provider through synchronous fusion of the inverse quantization results of the first sparse data, i.e., the sparse data fed back by other training members; the step of updating the first local model based on the first processing tensor further includes: replacing the local parameters to be synchronized with the synchronization tensor, thereby completing the synchronization update of the first local model with the local models held by other training members.

[0010] In one embodiment, the number of elements in the first intermediate tensor is n1, and the number of elements retained after sparsification is n2, where n2 is less than n1. The effective elements in the first sparsified tensor include a first element, which corresponds to a first index position and a first element value. The first index value is between 0 and n1. The step of quantizing the elements retained in the first sparsified tensor to obtain the first sparse data includes: mapping the first element value to a range of values ​​described by the number of bytes corresponding to a first numeric type to a range of values ​​described by the number of bytes corresponding to a second numeric type, thereby obtaining a first quantized value corresponding to the first element value; mapping the first index position to a range of values ​​described by the number of bytes, with the resulting integer serving as the first coordinate value of the first index and the resulting remainder serving as the second coordinate value of the first index, wherein the first coordinate value and the second coordinate value constitute the first index coordinates corresponding to the first element.

[0011] In one embodiment, the first coordinate value is described by a second numeric type under a second byte number, which is the number of bytes required to describe the minimum binary number of the first coordinate value, according to the obtained integer size.

[0012] In one embodiment, the first numeric type is floating-point and the second numeric type is integer.

[0013] According to the second aspect, a method for jointly training a model is provided, in which a service provider and multiple training members jointly update the model using their local privacy data, wherein the model corresponds to a local model for each training member and a global model for the service provider; the method is executed by the service provider, and during the current model update cycle, the method includes: receiving sparse tensors for corresponding intermediate tensors from each training member, wherein a single intermediate tensor is obtained by processing local feature data of several training samples in the current batch based on the corresponding local model; the single sparse tensor is sparsified by the corresponding training member for the corresponding single intermediate tensor to obtain a sparse tensor described by a first numerical type, and quantizing the elements retained in the sparse tensor, wherein the sparse data is described by a second numerical type for the corresponding elements, and the number of bytes corresponding to the first numerical type is greater than the number of bytes corresponding to the second numerical type; dequantizing each sparse data to recover each intermediate tensor; performing subsequent processing on each intermediate tensor to obtain each processed tensor corresponding to each intermediate tensor; and feeding back the corresponding processed tensor to each training member so that each training member can update its local model based on the corresponding processed tensor.

[0014] According to a third aspect, an apparatus for jointly training a model is provided, in which multiple training members of a service provider jointly update the model using their local privacy data. The multiple training members include a first member, which holds first feature data for training samples. The model has a first local model corresponding to the first member and a global model corresponding to the service provider. The apparatus is located at the first member and includes a processing unit, a sparsification unit, a quantization unit, a providing unit, and an updating unit. During the current model update cycle:

[0015] The processing unit is configured to process the first feature data of several training samples in the current batch based on the first local model to obtain a first intermediate tensor;

[0016] A sparsification unit is configured to sparsify the first intermediate tensor to obtain a first sparsified tensor with a first numerical type description.

[0017] The quantization unit is configured to quantize the elements retained in the first sparse tensor to obtain first sparse data, wherein the first sparse data describes the corresponding elements through a second numeric type, and the number of bytes corresponding to the first numeric type is greater than the number of bytes corresponding to the second numeric type.

[0018] The providing unit is configured to provide the service provider with first sparse data describing a second numerical type, so that the service provider can process the inverse quantization result of the first sparse data accordingly and then feed back a first processed tensor corresponding to the first intermediate tensor.

[0019] The update unit is configured to update the first local model based on the first processing tensor.

[0020] According to the fourth aspect, an apparatus for jointly training a model is provided, in which a service provider and multiple training members jointly update the model using their local privacy data, wherein the model has local models corresponding to each training member and a global model corresponding to the service provider; the apparatus is located on the service provider and includes a receiving unit, an inverse quantization unit, a processing unit, and a feedback unit, in the current model update cycle:

[0021] The receiving unit is configured to receive sparse tensors for corresponding intermediate tensors from each training member. The individual intermediate tensor is obtained by processing the local feature data of several training samples in the current batch based on the corresponding local model. The individual sparse tensor is sparsified by the corresponding training member for the corresponding individual intermediate tensor to obtain a sparse tensor described by a first numerical type. The elements retained in the sparse tensor are then quantized. The sparse data is described by a second numerical type for the corresponding elements. The number of bytes corresponding to the first numerical type is greater than the number of bytes corresponding to the second numerical type.

[0022] The dequantization unit is configured to dequantize each sparse data point to recover the intermediate tensors.

[0023] The processing unit is configured to perform subsequent processing on each intermediate tensor to obtain each processed tensor corresponding to each intermediate tensor.

[0024] The feedback unit is configured to feed back the corresponding processing tensor to each training member, so that each training member can update its local model based on the corresponding processing tensor.

[0025] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first or second aspect.

[0026] According to a sixth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect or the second aspect.

[0027] The method and apparatus provided in the embodiments of this specification, when a training member provides a corresponding intermediate tensor to the service provider, firstly, the intermediate tensor is sparsified, and then the effective elements after sparsification are quantized (reducing the number of bytes), thereby providing the sparsed data of the intermediate tensor to the service provider with a smaller amount of data. The service provider, upon receiving the sparsed data, first performs dequantization to recover the sparse tensor corresponding to the intermediate tensor, and then performs subsequent processing. Afterwards, the service provider feeds back the processed tensor corresponding to the intermediate tensor of the training member to each training member based on the subsequent processing, so that each training member can update their local model. This method further quantizes the sparsified intermediate tensor, greatly reducing the number of bytes in the values, thereby reducing the amount of communication data, shortening communication time, and improving model training efficiency. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of the implementation architecture of federated learning is shown;

[0030] Figure 2a A schematic diagram of the model architecture under horizontal federated learning is shown.

[0031] Figure 2b A schematic diagram of the model architecture under vertical federated learning is shown;

[0032] Figure 3 This diagram illustrates the interaction flow between the training member and the service provider in a joint training model process according to one embodiment.

[0033] Figure 4 A schematic block diagram of a joint training model apparatus for a single training member is shown according to one embodiment;

[0034] Figure 5 A schematic block diagram of a joint training model apparatus provided on a service provider according to one embodiment is shown. Detailed Implementation

[0035] The technical solutions provided in this specification are described below with reference to the accompanying drawings.

[0036] First, let's describe some concepts that may be involved in this instruction manual.

[0037] Federated learning, also known as federated machine learning, consortium learning, or alliance learning, is a machine learning framework that effectively helps multiple organizations use data and perform machine learning modeling while meeting user privacy, data security, and government regulations.

[0038] Specifically, suppose company A and company B each build a task model, where a single task could be classification or prediction, and these tasks have already been approved by their respective users when the data was acquired. However, due to incomplete data—for example, company A lacks label data, company B lacks user feature data, or the data is insufficient, with an inadequate sample size to build a good model—the models on each end may fail to be built or perform poorly. Federated learning aims to solve the problem of how to build high-quality models on both A and B, where the training of the model utilizes data from both companies, and each company's proprietary data remains unknown to other parties, i.e., a shared model is built without violating data privacy regulations. This shared model is like the optimal model built by aggregating the data from all parties. In this way, the built model serves only the respective objectives of each party in its region.

[0039] In federated learning, each institution can be referred to as a training member. Each training member can hold different business data and can participate in the joint training of the business model through devices, computers, servers, etc. This business data can be various types of data, such as characters, images, voice, animation, and video. Typically, the business data held by each training member is correlated, and the business entities corresponding to each training member can also be correlated. For example, among multiple business entities involved in financial business, business entity 1 is a bank, providing savings and loan services to users, and can hold data such as users' age, gender, income and expenditure records, loan amounts, and deposit amounts; business entity 2 is an investment and wealth management platform, and can hold data such as users' loan records, investment records, and repayment deadlines; business entity 3 is a shopping website, and can hold data such as users' shopping habits, payment habits, and payment accounts. For example, in a healthcare business involving multiple stakeholders, each stakeholder could be a hospital, a medical examination institution, etc. Stakeholder 1 could be Hospital A, whose local business data includes user age, gender, symptoms, diagnosis results, treatment plans, treatment outcomes, etc. Stakeholder 2 could be Medical Examination Institution B, whose local business data includes user age, gender, symptoms, examination conclusions, etc., and so on. A single training member can hold business data from one stakeholder or multiple stakeholders. The purpose of federated learning is to train a model that can better handle this business data; therefore, a federated learning model can also be called a business model.

[0040] Figure 1This illustrates a specific implementation architecture of federated learning. Under this architecture, multiple training members (such as member 1, member 2, member 3, etc.) can process their locally held feature data through their local models to obtain corresponding intermediate tensors. Each training member can provide these intermediate tensors to the service provider, which, based on the fusion processing of these intermediate tensors, feeds back the corresponding processed tensors to each training member, allowing them to update their local models accordingly.

[0041] Federated learning is divided into horizontal federated learning and vertical federated learning. Figure 2a Figure 2 illustrates the model architecture of horizontal federated learning. As shown in Figure 2, in this architecture, the sample sets of different training members have a high degree of feature overlap, but the sample sources are different. For example, multiple sample sets correspond to customers of different banks. Generally, the data features managed by banks are similar, but the customers are different; thus, horizontal federated learning can be used to train the model.

[0042] In vertical federated learning, different datasets often have high overlap in IDs (e.g., identical phone numbers), but different features. For example, consider a bank and a hospital serving the same user group (e.g., residents of a small county). The bank and hospital samples have a high degree of overlap in individuals, but their features differ. Bank data might correspond to features like deposits and loans, while hospital data might correspond to features like physiological indicators, health status, and medical records. Training a model using the combined bank and hospital datasets is called vertical federated learning. In horizontal federated learning, each training member typically holds a local model. Each local model has the same structure as the global model. During the training of their local models, each member updates synchronously with the global model. The synchronous parameter values ​​of the global model are obtained by fusing the processing results of each local model. The global model can exist physically or as synchronous parameters. The synchronous fusion of the global model is usually performed by a trusted third party acting as the service provider.

[0043] Figure 2b A diagram illustrating the model architecture of vertical federated learning is shown. Figure 2bAs shown, in a vertical federated learning scenario with vertically segmented data, the business model is typically divided into two parts: a local model held by each training member to process local feature data and obtain intermediate results, and a global model to process the intermediate results from each training member and obtain the final output. In this scenario, multiple training members can also use a decentralized multi-party secure computation (MPC) architecture, where the global model can be distributed among the training members holding labeled data. Alternatively, a centralized third-party auxiliary architecture can be adopted, with the third party acting as a service center (or service provider). In this case, the global model can be located on the service provider. Each training member must provide the intermediate data after feature data processing to the service provider, and the training member holding labeled data must also provide the labeled data to the service provider.

[0044] In this context, the service provider can be a trusted third party or a label holder in vertical federated learning. It is assumed that the service provider holds the global model. Those skilled in the art will readily understand that in vertical federated learning, a single training member can send the service provider intermediate results obtained by processing partial features of the locally held training samples using their local model. In horizontal federated learning, a single training member can send the parameters to be synchronized to the service provider. Regardless of whether it's horizontal or vertical federated learning, in a model update cycle, each training member sends the processed results of its local data to the service provider at least once. In this specification, the processed results of the local data sent by a single training member to the service provider can be referred to as intermediate tensors.

[0045] In a horizontal federated learning architecture, each member holds complete feature data of the training samples. Predictions are obtained by processing this feature data, and the model loss is derived by comparing the predictions with the label data. The gradients of each undetermined parameter are then determined based on the model loss. However, in a horizontal federated learning architecture, the parameters to be synchronized (i.e.,...) Figure 1 The intermediate tensor (in the model) can be at least one of the gradient of each undetermined parameter, the change in each undetermined parameter, or the updated value of each undetermined parameter. The change in each undetermined parameter and the updated value of each undetermined parameter can both be determined by the gradient of each undetermined parameter. In a longitudinal federated learning architecture, a single training member can hold partial feature data of the training samples, and the intermediate result can be the processing result of this partial feature data by the local model.

[0046] Combination Figure 1 , Figure 2a , Figure 2bAs shown, each training member can pass its corresponding intermediate tensors to the server. In the vertical federated learning architecture, the server can fuse the intermediate tensors and process them through the global model to obtain the prediction results, thereby determining the model loss and backpropagating the gradient. During the backpropagation gradient process, the server determines and updates the gradients of each undetermined parameter of the global model based on the model loss, and also feeds back the gradient of the model loss with respect to the corresponding intermediate tensor to each training member. Each training member can determine the gradient of each undetermined parameter in its local model based on the gradient of the intermediate vector, and use it to update its local model. In the horizontal federated learning architecture, the server can synchronously fuse the intermediate tensors of each training member, for example, taking the mean, maximum, minimum, median, etc., as synchronization values. Then, the server feeds back the synchronization values ​​corresponding to each undetermined parameter to each training member, and each training member uses the synchronization values ​​to update the values ​​of the undetermined parameters, thereby updating the model.

[0047] As can be seen, in the federated learning process, for a single model update cycle, a single training member communicates with the server at least twice: uploading intermediate tensors and receiving gradients or synchronization values ​​determined based on the intermediate tensors. To reduce data communication volume, shorten communication time, and improve federated learning efficiency, this specification provides a data transmission method based on data quantization. This method further reduces the number of bytes by quantizing the elements retained after sparsification, based on the sparsification of the intermediate tensors, thereby reducing the data transmission volume.

[0048] The following is for reference. Figure 3 A specific example is shown to illustrate the technical concept of this specification.

[0049] like Figure 3 The diagram illustrates a process for jointly training a model according to one embodiment. This process is applicable to either vertical federated learning or horizontal federated learning. The implementation architecture of this process may include multiple training members and service providers. As described above, depending on the actual form of the federated learning architecture, the service provider may be a trusted third party or a training member holding labeled data under a vertical federated learning architecture. Figure 3The illustrated process describes the interaction between the first member among the training members and the service provider. The first member can be any one of the multiple training members. The first member can hold the first local model, and the service provider can hold the global model. In the horizontal federated learning process, the first local model and the global model have the same structure. The first local model can independently process feature data, obtain prediction results, determine the model loss, and backpropagate the gradients of the undetermined parameters on the first member. The global model can be used to synchronously fuse the local models of each training member. The global model may not actually exist, but is only described as the synchronization value of the parameters to be synchronized by the service provider. In the vertical federated learning architecture, the global model can be connected after the local models of each training member. It fuses and further processes the intermediate results output by each local model to obtain the prediction result, which the service provider compares with the prediction result of the global model and the label data, thereby backpropagating the gradient for training the global model and each local model.

[0050] The embodiments in this specification are based on Figure 1 The illustrated architecture is improved by sparsifying and quantizing the intermediate tensors when a single training member provides them to the service provider, thereby reducing the amount of data provided to the service provider while ensuring data privacy. See below for reference. Figure 3 The interactive flow shown describes how the first member and the service provider process the data during a model update cycle.

[0051] First, in step 301, the first member processes the first feature data of several training samples in the current batch based on the first local model to obtain the first intermediate tensor (denoted as M1).

[0052] Here, the first intermediate tensor can be a one-dimensional tensor, a two-dimensional tensor, or a three-dimensional tensor; this specification does not limit it. For example, in a vertical federated learning architecture, the first local model can obtain an m-dimensional embedding vector for each training sample. n training samples can yield n m-dimensional embedding vectors, for example, forming an n×m two-dimensional tensor as the first intermediate tensor. For ease of description, the number of elements in the first intermediate tensor can be denoted as n1, where n1 is a positive integer greater than 1.

[0053] It is understood that the first member holds at least several sets of first feature data for training samples. Each set of first feature data can correspond to one or more feature items. The first feature data can be extracted in advance from local data. For example, if one training sample corresponds to one user, and the local data held by the first training member consists of the user's financial management, loan, and repayment data, then features such as financial management type, financial management amount, financial management return, loan frequency, loan amount, and repayment timeliness can be extracted as the first feature data for the corresponding training sample. The current batch of training samples can include one or more training samples.

[0054] In a vertical federated learning architecture, several training samples in the current batch can be sampled from the local dataset by each training member through a consensus-based, privacy-preserving method, and the sampling results of each training member are aligned with each other. The first feature data can be a subset of the feature data of these training samples. For example, if a training sample has 100 feature items, and the first member holds 10 of them, these 10 feature items can be called the first feature data of the training sample. Correspondingly, the first intermediate tensor can be the processing result of the first local model on these 10 feature items (first feature data) of the current batch. At this time, the first local model can be an embedding model or an encoding model, which fuses the feature values ​​in the first feature data or mines its deep features and represents them through vectors of predetermined dimensions.

[0055] In a horizontal federated learning architecture, the training samples in the current batch can be sampled independently by each training member from their local training samples. In this case, the first feature data can be all the feature data from several training samples sampled by the first member. For example, if a training sample has 100 feature items and a label, the first member can hold 100 feature items and a label from each of several training samples. The first local model can then independently process the feature data and obtain the prediction result. The first member can also compare the corresponding prediction result with the label data to obtain the model loss for the current update cycle. Furthermore, to reduce the model loss, the gradient of the undetermined parameter W can be determined according to the structure of the first local model. For example, if the first local model is a multi-layer fully connected neural network, and the expression for a single-layer neural network is y = Wx + b, then the gradient of the undetermined parameter W is, for example: Where L is the model loss, x is the partial derivative of y with respect to W, and when calculating the gradient of the undetermined parameter W, x can be the actual input value during the processing of the current batch of training samples. In this case, the first intermediate tensor can be the parameters to be synchronized in the local models of each training member. This is especially relevant when the undetermined parameters are based on gradients (e.g., W = W - γ·G). w The parameter to be synchronized can be the gradient of the parameter itself (e.g., G).w It can also be the change of an undetermined parameter (such as -γ·G). w It can also be the updated value of the undetermined parameter (i.e., the updated value, such as W = W - γ·G). w In an alternative embodiment, during a synchronization cycle of the model, a single training member can update the local parameters to be determined in one or more rounds.

[0056] Regardless of whether it's horizontal or vertical federated learning, the first training member can provide a first intermediate tensor to the service provider. Under the technical concept described in this specification, the first intermediate tensor is reduced in size to decrease communication overhead.

[0057] Therefore, in step 302, the first member sparsifies the first intermediate tensor to obtain the first sparsified tensor (denoted as M). 1p ).

[0058] The sparsification process involves setting some elements to 0. In other words, some elements are retained as valid elements, which can be called valid elements. Assuming the number of valid elements is n2, then n2 is less than n1, and the remaining elements are set to 0. Sparsification of the first intermediate tensor can be performed using various sparsification methods such as random sparsification and Top K sparsification. Taking Top K as an example, the K largest elements are retained (where n2 = K), and the other values ​​are set to 0. Here, K is a preset value, or an integer value determined according to the sparsity rate p, which is not limited here. The sparsified result of the first intermediate tensor can be called the first sparsified tensor.

[0059] In step 303, the first member quantizes the elements retained in the first sparsified tensor to obtain the first sparse data (denoted as M'). 1p ).

[0060] Quantization is a method of information compression. In machine learning, quantization can be seen as a readjustment of the range of numerical values, for example, adjusting values ​​from the range of 0-1.0 to the range of 0-255. When machine language uses 0 and 1 to represent data, this adjustment of the numerical range can be described by changes in the numeric type and the number of bytes occupied. A numeric type is a set of values ​​with the same properties defined in a data structure, along with a set of operations performed on that set of values. Numeric types can include, but are not limited to, integers, floating-point numbers, characters, and booleans.

[0061] Taking floating-point numbers as an example, floating-point numbers are numerical representations of numbers belonging to a specific subset of rational numbers, used in computers to approximate any real number. Specifically, this real number is obtained by multiplying an integer or fixed-point number (i.e., the mantissa) by an integer power of a base (usually 2 in computers). In C++, floating-point numbers can include, for example, 32-byte single-precision floating-point numbers (float), 32-byte single-precision unsigned floating-point numbers (unsigned float), 64-byte double-precision floating-point numbers (double), 80-byte high-precision double-precision floating-point numbers (long double), and so on.

[0062] In the field of machine learning, to improve computational precision, 64-byte double-precision floating-point numbers (double) are typically used to represent data. When dealing with large amounts of data, if each data point is 64 bytes (e.g., it can represent -2...), then... 63 Up to 2 63 Numbers between -1 and 1 require a relatively large amount of communication. To reduce communication, values ​​can be represented as other numeric types, such as integers. Integer types include byte, short, int, and long. Integer values ​​have a relatively small number of bytes, such as 8 bytes (for representing numbers between 0-255 or -128 to 127).

[0063] In other words, the quantization process described in this specification is the conversion of a high-byte numeric type into a low-byte numeric type. Quantization can be performed through linear or non-linear mapping, preserving the numerical value while accepting some information loss, representing the value as a numeric type with fewer bytes. This specification uses the mapping from floating-point to integer as an example. In practice, other high-byte numeric types can also be converted to other low-byte numeric types. Taking the linear mapping from floating-point to integer as an example, a floating-point value 'a' between 0 and 1.0 is mapped to an 8-byte integer value 'b' between 0 and 255. The mapping process can be b = 255a. For data precision, floating-point data may have some bytes representing the decimal part. During the numerical range conversion process, there may be a possibility that the mapped value has a decimal part, while the integer value does not. The decimal part needs to be sacrificed, such as through rounding, rounding up, or rounding down. This loss of the decimal part is the information loss during data type conversion. If a = 0.0159, then b = 255 and a = 4.0545. We can round this to the nearest integer 4 as the quantized integer value, for example, representing it as an 8-byte integer value of 00000100. Considering that gradients or undetermined parameters are usually signed numbers, floating-point numbers can also be between -0.5 and 0.5. In this case, int 8 can be defined as a number between -128 and 127. The conversion method is similar to that described earlier and will not be repeated here.

[0064] In practice, when sending a sparse matrix, only the valid values—those retained during the sparsification process—can be sent. For example, for an n×m matrix, the retained values ​​include the value in the 2nd row and 10th column. To describe the position of the retained elements, for a single element j, it can correspond not only to the numerical value Value... j It can also correspond to an index. j Among them, the element value is Value. j Numeric type conversions can be performed using the mapping methods described above. And the index... j Typically positive numbers, using unsigned integer values ​​can save bytes when describing indices. For example, an 8-byte unsigned integer (such as uint 8) can describe values ​​between 0 and 255.

[0065] When dealing with large datasets, the number of rows and columns in the index data can be substantial. Using the original row and column counts directly might also result in large values. For example, if the feature count *m* is 10,000 and the current batch sample size is 100, then the total number of rows and columns (n×m) would be 100×10,000. This would require 15 bytes of unsigned integer data to describe the number of columns, which is still quite large.

[0066] Therefore, according to one possible design, the position of each retained element in the first sparse tensor within the total number of elements can be identified first. Assuming the total number of elements in the first intermediate tensor before sparsification is n1, each element can be identified sequentially using values ​​between 0 and n1. Then, the identifiers of the elements retained during the sparsification process are converted into two index coordinates to describe the element's index position.

[0067] In one optional implementation, a first byte size c (usually a multiple of 2, such as 2, 4, 8, 16, etc.) of unsigned integer data, such as uint 8 (i.e., 8 bytes of unsigned integer data), can be defined as the first quantization bit depth, and the index coordinates of the reserved elements can be determined according to this predetermined number of bytes of unsigned integer data. First, the first byte size c of unsigned integer data is 0 to N (e.g., N=2). c If -1 = 255, then division with N can be performed using the index position identifier of the reserved element. The resulting integer quotient and remainder can both be represented by unsigned integer data of the first byte c between 0 and N, thus each serving as an index coordinate of the corresponding element. For example, Index j1 =round(Index j / N), Index j2 =(Index j )mod N. Where round means rounding down (rounding down), and mod means modulo.

[0068] Typically, the remainder ranges from 0 to N, while the integer part of the quotient has a larger range. For example, 2 can be represented as the binary number 10 using 2 bytes, 14 can be represented as the binary number 1110 using 4 bytes, and 32 may be represented as the binary number 100000 using 6 bytes. If a uniform quantization bit size (e.g., 8 bytes for the first byte) is used, then 2 needs to be described as 00000010. With a first byte size of 8, the number of elements with a quotient of 2 can be at most 256. This still results in a lot of byte redundancy. Therefore, to further save bytes, in one embodiment of this specification, the number of the second byte can be further determined based on the size of the quotient. This second byte size can be a dynamic bit size, which can be the smallest number of bytes that can describe the integer part of the quotient in binary. For example, an element identified as 10000, when quantized with a first byte size c = 8, is 10000 divided by (2^10 / 2^10). 8 The integer part of -1), i.e., 39, can be represented by the 6-byte binary number 100111 (in this case, the second byte is 6). Elements identified between 512 and 102 have their identifiers divided by (2...). 8 The integer part of -1 is 2 or 3, which can be represented by a 2-byte binary number 10 or 11 (in which case the second byte is 2). Thus, by defining a new number of bytes based on the size of the integer part, each retained element can be quantized according to the flexible number of bytes, thereby reducing byte redundancy and effectively describing each element in the sparse tensor.

[0069] After quantizing the elements in the first sparse tensor, a single retained element (an element not set to 0) corresponds to two index coordinates and one quantized numerical result, for a total of three values. Elements set to 0 do not correspond to any numerical value. The total number of bytes for the two index coordinates and the quantized numerical result is also significantly reduced compared to before quantization. For example, before quantization, it is 32 or 64 bytes of floating-point data, while after quantization, it is 8 + 2 + 8 = 18 bytes of integer data. The data after quantization of the first sparse tensor can be called first sparse data. This first sparse data retains only the data of the valid elements, and the number of bytes describing the data is greatly reduced compared to before quantization.

[0070] Each training member can determine the sparse data of the corresponding intermediate tensor in the above manner.

[0071] Step 304: The first member provides the service provider with the first sparse data, which is described by the second numerical type. Depending on the federated learning architecture, the service provider here may be a trusted third party or a training member holding the labels. In the vertical federated learning architecture, the first member is not a training member holding the labeled data.

[0072] Step 305: After processing the inverse quantization result of the first sparse data, the service provider returns the first processed tensor corresponding to the first intermediate tensor.

[0073] The service provider's dequantization operation on the first sparse data is the operation to recover the first sparse tensor. That is, based on the index coordinates of each element... j1 Index j2 ) and numerical quantization results (Value j The dequantization process reconstructs the first sparse tensor describing each element in the corresponding position, using the first numeric type (such as floating-point). Dequantization is the inverse of quantization; for example, it maps an integer range of 0-255 to a floating-point range of 0-1.0, which will not be elaborated further here. It's understandable that data with more bytes often contains decimals, so dequantization converts the lower-byte numeric type to the higher-byte numeric type, allowing for lossless information recovery. Dequantization can map data described by fewer bytes to data described by more bytes, thereby improving data processing accuracy.

[0074] The service provider can continue to perform relevant data processing on the first sparse tensor under the first numerical type.

[0075] In a horizontal federated learning architecture, the valid elements indicated in the first sparse data are the parameters to be synchronized. The service provider can recover each sparse tensor from the sparse data sent by each training member and perform synchronization fusion on these tensors. Synchronization fusion methods can include, for example, taking the mean, maximum, minimum, median, weighted sum, etc. Taking the mean as an example, for a single element, the service provider can add the valid values ​​of that single element fed back from each sparse tensor and divide by the number of valid values ​​to obtain the corresponding mean as the synchronization value of that single element. The synchronization values ​​of each element constitute a synchronization tensor and serve as the corresponding parameter values ​​of the global model. This synchronization tensor has the same meaning as the first intermediate tensor, such as the gradient, change, or update value of the parameters to be synchronized. In this specification, this synchronization tensor can be referred to as the first processing tensor, which is consistent with the first intermediate tensor. The service provider can feed back this first processing tensor to each training member.

[0076] In the vertical federated learning architecture, the effective elements indicated in the first sparse data are the processing results (such as embedding results) of each training sample by the first local model. The service provider can fuse the processing results of each training member on the local feature data (such as concatenation, summation, averaging, etc.) and further process the fused results using the global model to obtain the corresponding prediction results. Further, the overall model loss (including each local model and the global model) is determined based on the comparison between the prediction results and the label data. Then, based on the model loss, the service provider can determine the gradients of each undetermined parameter in the global model to update the undetermined parameters in the global model, and also determine the gradients of the model loss with respect to each intermediate tensor. The service provider can provide the gradients of the model loss with respect to each intermediate tensor to each training member. For the first member, the service provider can provide the gradient of the model loss with respect to the first intermediate tensor. Since this gradient is the gradient of the first intermediate tensor, it can be called the first processing tensor corresponding to the first intermediate tensor.

[0077] Thus, in either horizontal or vertical federated learning architectures, the service provider can provide feedback to the first member on the first processing tensor corresponding to the first intermediate tensor.

[0078] According to an optional embodiment, when the service provider returns the first processed tensor, it can also combine the sparsification, quantization, and dequantization processes. In this case, the returned data is the third sparsified data, which is the data after sparsification and quantization of the first processed tensor. This will not be elaborated here.

[0079] Step 306: The first member updates the first local model based on the first processing tensor.

[0080] If the service provider provides the aforementioned third sparsified data, the first member can first perform an inverse quantization operation on the third sparsified data to recover the first sparsified processing tensor.

[0081] In the horizontal federated learning architecture, the first member can directly use the corresponding elements in the first processing tensor to adjust the relevant undetermined parameters in the first local model, thereby updating the first local model.

[0082] In the longitudinal federated learning architecture, the first member can determine the gradient of the model loss with respect to each undetermined parameter in the first local model by using the gradient of the model loss with respect to the first intermediate tensor and the gradient of the first intermediate tensor with respect to each undetermined parameter in the first local model, based on the backpropagation property of the gradient, thereby updating each undetermined parameter in the first local model.

[0083] Reviewing the above process, the technical concept provided in this specification involves first sparsifying the intermediate tensors provided by training members to the service provider. Then, the effective elements after sparsification undergo quantization to reduce the number of bytes, thus providing the sparsed data of the intermediate tensors to the service provider with a smaller data volume. The service provider, upon receiving the sparsed data, first performs dequantization to recover the sparse tensor corresponding to the intermediate tensor and performs subsequent processing. Afterward, based on the subsequent processing, the service provider feeds back the processed tensor corresponding to the intermediate tensor of each training member to each training member, allowing each training member to update their local model. This method further quantizes the sparsified intermediate tensors, significantly reducing the number of bytes in the values, thereby reducing the amount of communication data, shortening communication time, and improving model training efficiency.

[0084] According to another embodiment, a joint training model apparatus is also provided. This apparatus is used by a service provider and multiple training members, each utilizing their local privacy data, to jointly update the model. This apparatus can be of two types: one is a device set up in the training members' area, such as... Figure 4 The illustrated device 400 is another type of device located on the server side, such as... Figure 5 The device 500 is shown. The two devices work together to complete the joint update of the model.

[0085] Specifically, Figure 4 An embodiment of a joint training model apparatus 400 is shown, which can be configured with a first member among a plurality of training members. For example... Figure 4 As shown, the device includes: a processing unit 401, a sparsification unit 402, a quantization unit 403, a providing unit 404, and an update unit 405. In the current model update cycle:

[0086] The processing unit 401 is configured to process the first feature data of several training samples in the current batch based on the first local model to obtain the first intermediate tensor;

[0087] The sparsification unit 402 is configured to sparsify the first intermediate tensor to obtain a first sparsified tensor with a first numerical type description.

[0088] Quantization unit 403 is configured to quantize the elements retained in the first sparse tensor to obtain first sparse data, wherein the first sparse data describes the corresponding elements through a second numeric type, and the number of bytes corresponding to the first numeric type is greater than the number of bytes corresponding to the second numeric type.

[0089] The providing unit 404 is configured to provide the service provider with the first sparse data describing the second numerical type, so that the service provider can process the inverse quantization result of the first sparse data accordingly and then feed back the first processed tensor corresponding to the first intermediate tensor.

[0090] Update unit 405 is configured to update the first local model based on the first processing tensor.

[0091] Figure 5 An embodiment of a joint training model apparatus 500 is shown, which can be configured on a service provider. For example... Figure 5 As shown, the device 500 includes a receiving unit 501, an inverse quantization unit 502, a processing unit 503, and a feedback unit 504. In the current model update cycle:

[0092] The receiving unit 501 is configured to receive sparse tensors for corresponding intermediate tensors from each training member. The single intermediate tensor is obtained by processing the local feature data of several training samples in the current batch based on the corresponding local model. The single sparse tensor is sparsified by the corresponding training member for the corresponding single intermediate tensor to obtain a sparse tensor described by a first numerical type. The elements retained in the sparse tensor are then quantized. The sparse data is described by a second numerical type for the corresponding elements. The number of bytes corresponding to the first numerical type is greater than the number of bytes corresponding to the second numerical type.

[0093] The dequantization unit 502 is configured to dequantize each sparse data to recover each intermediate tensor;

[0094] The processing unit 503 is configured to perform subsequent processing on each intermediate tensor to obtain each processed tensor corresponding to each intermediate tensor.

[0095] Feedback unit 504 is configured to feed back the corresponding processing tensor to each training member, so that each training member can update the local model based on the corresponding processing tensor.

[0096] It is worth noting that, Figure 4 , Figure 5 The devices 400 and 500 shown are respectively with Figure 3 The operations performed by the first party and the service provider in the described method correspond to each other. Figure 3 The corresponding descriptions in the method embodiments also apply to devices 400 and 500, and will not be repeated here.

[0097] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 The method executed by either party as described above.

[0098] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, it implements a combination... Figure 3The method executed by either party as described above.

[0099] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0100] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the technical concept of this specification. It should be understood that the above are merely specific embodiments of the technical concept of this specification and are not intended to limit the scope of protection of the technical concept of this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this specification should be included within the scope of protection of the technical concept of this specification.

Claims

1. A method for jointly training a model, wherein a service provider and multiple training members jointly update the model using their local privacy data, wherein, The plurality of training members includes a first member, which holds first feature data for the training samples. The model has a first local model corresponding to the first member and a global model corresponding to the service provider. The method is executed by the first member. In the current model update cycle, the method includes: Based on the processing of the first feature data of several training samples in the current batch by the first local model, the first intermediate tensor is obtained. Sparsify the first intermediate tensor to obtain a first sparsified tensor with a first numerical type description. The elements retained in the first sparse tensor are quantized to obtain the first sparse data. The first sparse data describes the corresponding elements through a second numeric type. The number of bytes corresponding to the first numeric type is greater than the number of bytes corresponding to the second numeric type. The quantization process converts the representation of the elements from a high-byte numeric type to a low-byte numeric type to reduce the amount of communication data and shorten the communication time. The service provider is provided with first sparse data describing a second numerical type, so that the service provider can process the inverse quantization result of the first sparse data accordingly and then return a first processed tensor corresponding to the first intermediate tensor. The inverse quantization result is determined based on the conversion of the element representation from a low-byte numerical type to a high-byte numerical type. The first local model is updated based on the first processing tensor.

2. The method as described in claim 1, wherein, In vertical federated learning, the service provider is a training member or a trusted third party holding labeled data. The global model is sequentially connected to the first local model, and the first intermediate tensor is the processing result of the first local model on the first feature data of the several training samples.

3. The method as described in claim 2, wherein, The first processing tensor is: the first intermediate gradient of the model loss with respect to the first intermediate tensor, wherein the first intermediate gradient is obtained by the service provider based on the dequantization of the first sparse data, the second intermediate tensor is processed by the global model to obtain the fusion result of the second intermediate tensor and other intermediate tensors fed back by other training members, and the corresponding prediction result is obtained. The model loss is determined by comparing the prediction result with the label data, thereby determining the first intermediate gradient through gradient backpropagation. The step of updating the first local model based on the first processing tensor further includes: The first intermediate gradient is used to determine the gradients corresponding to each undetermined parameter in the first local model. The first local model is updated by updating each undetermined parameter in the first local model according to each gradient.

4. The method of claim 1, wherein, In horizontal federated learning, the service provider is a trusted third party, the global model has the same structure as the first local model, and the first intermediate tensor is a tensor used to describe the parameters to be synchronized. The parameters to be synchronized are one of the following: the change of the parameters to be determined, the gradient of the parameters to be determined, and the updated value of the parameters to be determined.

5. The method of claim 1, wherein, The first processing tensor is: a fusion tensor obtained by the service provider through synchronous fusion of the inverse quantization results of the first sparse data, i.e., the sparse data fed back by other training members. The step of updating the first local model based on the first processing tensor further includes: The local parameters to be synchronized are replaced by the first processing tensor, thereby completing the update of the synchronization between the first local model and the local models held by other training members.

6. The method of claim 1, wherein, The number of elements in the first intermediate tensor is n1, and the number of valid elements retained after sparsification is n2, where n2 is less than n1. The valid elements in the first sparsified tensor include a first element, which corresponds to a first index position and a first element value. The first element value takes a value between 0 and n1. The step of quantizing the elements retained in the first sparse tensor to obtain the first sparse data includes: The first element value is rounded and mapped to the value range described by the number of bytes corresponding to the first numeric type and then to the value range described by the number of bytes corresponding to the second numeric type to obtain the first quantized value corresponding to the first element value. The first index position is mapped according to the value range of the third numeric type described by the first byte number, and the resulting integer is used as the first coordinate value of the first index, and the resulting remainder is used as the second coordinate value of the first index, wherein the first coordinate value and the second coordinate value constitute the first index coordinates corresponding to the first element.

7. The method of claim 6, wherein, The first coordinate value is described by a third numeric type under the second byte number, which is the minimum number of bytes used to describe the first coordinate value. The third numeric type is an unsigned integer.

8. The method of claim 1, wherein, The first numeric type is floating-point, and the second numeric type is integer.

9. A method for jointly training a model, wherein a service provider and multiple training members jointly update the model using their local privacy data, wherein, The model has local models corresponding to each training member and a global model corresponding to the service provider; the method is executed by the service provider, and in the current model update cycle, the method includes: The system receives sparse tensors for corresponding intermediate tensors from each training member. Each intermediate tensor is obtained by processing local feature data of several training samples in the current batch based on the corresponding local model. Each sparse tensor is sparsified by the corresponding training member to obtain a sparse tensor described by a first numerical type. The elements retained in the sparse tensor are then quantized. The sparse data is described by a second numerical type. The number of bytes corresponding to the first numerical type is greater than the number of bytes corresponding to the second numerical type. The quantization process converts the representation of the elements from a high-byte numerical type to a low-byte numerical type to reduce the amount of communication data and shorten the communication time. Each sparse data is dequantized to recover each intermediate tensor. The dequantization is based on the conversion of the element representation from a low-byte numeric type to a high-byte numeric type. Each intermediate tensor is then processed to obtain the corresponding processed tensor. Each training member is fed back the corresponding processing tensor so that each training member can update its local model based on the corresponding processing tensor.

10. An apparatus for jointly training a model, wherein a service provider and multiple training members jointly update the model using their respective local privacy data, wherein, The plurality of training members includes a first member, which holds first feature data for the training samples. The model has a first local model corresponding to the first member and a global model corresponding to the service provider. The device is located in the first member and includes a processing unit, a sparsification unit, a quantization unit, a providing unit, and an update unit. In the current model update cycle: The processing unit is configured to process the first feature data of several training samples in the current batch based on the first local model to obtain a first intermediate tensor; A sparsification unit is configured to sparsify the first intermediate tensor to obtain a first sparsified tensor with a first numerical type description. The quantization unit is configured to quantize the elements retained in the first sparse tensor to obtain first sparse data. The first sparse data describes the corresponding elements through a second numeric type. The number of bytes corresponding to the first numeric type is greater than the number of bytes corresponding to the second numeric type. The quantization process converts the representation of the elements from a high-byte numeric type to a low-byte numeric type to reduce the amount of communication data and shorten the communication time. The providing unit is configured to provide the service provider with first sparse data describing a second numerical type, so that the service provider can process the inverse quantization result of the first sparse data accordingly and then feed back a first processed tensor corresponding to the first intermediate tensor. The inverse quantization result is determined based on the conversion of the element representation from a low-byte numerical type to a high-byte numerical type. The update unit is configured to update the first local model based on the first processing tensor.

11. An apparatus for jointly training a model, wherein a service provider and multiple training members jointly update the model using their respective local privacy data, wherein, The model has local models corresponding to each training member and a global model corresponding to the server; the device is located on the server and includes a receiving unit, an inverse quantization unit, a processing unit, and a feedback unit, in the current model update cycle: The receiving unit is configured to receive sparse tensors for corresponding intermediate tensors from each training member. Each intermediate tensor is obtained by processing local feature data of several training samples in the current batch based on the corresponding local model. Each sparse tensor is sparsified by the corresponding training member to obtain a sparse tensor described by a first numerical type. The elements retained in the sparse tensor are then quantized. The sparse data is described by a second numerical type. The number of bytes corresponding to the first numerical type is greater than the number of bytes corresponding to the second numerical type. The quantization process converts the representation of the elements from a high-byte numerical type to a low-byte numerical type to reduce the amount of communication data and shorten the communication time. The dequantization unit is configured to dequantize each sparse data to recover each intermediate tensor, wherein the dequantization is based on the conversion of the element representation from a low-byte numeric type to a high-byte numeric type; The processing unit is configured to perform subsequent processing on each intermediate tensor to obtain each processed tensor corresponding to each intermediate tensor. The feedback unit is configured to feed back the corresponding processing tensor to each training member, so that each training member can update its local model based on the corresponding processing tensor.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.

13. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-9.

Images