Federal long tail learning method based on synthetic pseudo features and data processing system

Abstract

The invention provides a federal long tail learning method based on synthetic pseudo features and a data processing system, the data processing system comprises a server and at least two clients, the server splits a business processing model to obtain an encoder and sends encoder parameters of the encoder to each client; the at least two clients are used for processing the local service data based on the encoder parameters to obtain local service feature statistics and sending the local service feature statistics to the server; the server is used for aggregating the local service feature statistical magnitude of each client into a global service feature statistical magnitude, determining a target pseudo feature based on the global service feature statistical magnitude and the initial pseudo feature, and adjusting a classifier in a service processing model according to the target pseudo feature to obtain a target service processing model, the target business processing model is used for classifying the target object. And the classifier is adjusted through the target false features, so that the classification accuracy of the business processing model on the target object is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a federated long-tail learning method based on synthetic pseudo features. The present invention also relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous advancement of machine learning and artificial intelligence technology, dealing with category imbalance problems, especially the long-tail data phenomenon, has become one of the key challenges to improving model performance. In the federated learning environment, due to its distributed nature and the need for privacy protection, solving the long-tail data problem is particularly complex and important.

[0003] At present, the processing of long-tail data mainly focuses on adjusting loss functions, resampling strategies (such as oversampling minority classes or undersampling majority classes), using cost-sensitive learning algorithms and other methods to improve the generalization ability of the model. In federated learning, researchers have explored personalized learning, transfer learning, and designing new aggregation mechanisms to adapt to non-IID data distribution, trying to alleviate global and local class imbalance problems.

[0004] Despite the above efforts, existing technologies still have some limitations. In a federated learning environment, traditional centralized long-tail learning solutions are difficult to apply directly due to the non-independent and identically distributed (non-IID) of data and privacy protection restrictions. In addition, existing methods often fail to fully consider the coexistence of global and local imbalances in federated learning, as well as the additional challenges brought by mismatch imbalance, resulting in difficulties in global model convergence, increased deviations, and decreased performance. Therefore, how to improve the processing performance of business processing models under federated learning, and then improve the classification accuracy of business processing models, has become a problem that needs to be solved urgently by technicians in this field. Summary of the invention

[0005] In view of this, an embodiment of this specification provides a federated long-tail learning method based on synthetic pseudo features. One or more embodiments of this specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a federated long-tail learning method based on synthetic pseudo features is provided, which is applied to a data processing system, wherein the data processing system includes a server and at least two clients, wherein: The server splits the service processing model to obtain an encoder, and sends encoder parameters of the encoder to each client; The at least two clients process the local service data based on the encoder parameters to obtain local service feature statistics, and send the local service feature statistics to the server; The server aggregates the local business feature statistics of each client into global business feature statistics, determines the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjusts the classifier in the business processing model according to the target pseudo-feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object.

[0007] According to a second aspect of an embodiment of this specification, a data processing system is provided, wherein the data processing system includes a server and at least two clients, wherein: The server is configured to split the service processing model to obtain an encoder, and send encoder parameters of the encoder to each client; The at least two clients are configured to process the local service data based on the encoder parameters to obtain local service feature statistics, and send the local service feature statistics to the server; The server is configured to aggregate the local business feature statistics of each client into a global business feature statistics, determine the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjust the classifier in the business processing model according to the target pseudo-feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object.

[0008] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned federated long-tail learning method based on synthetic pseudo features are implemented.

[0009] According to a fourth aspect of an embodiment of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned federated long-tail learning method based on synthetic pseudo-features are implemented.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned federated long-tail learning method based on synthetic pseudo-features.

[0011] An embodiment of the present specification realizes that the server splits the business processing model to obtain an encoder, and sends the encoder parameters of the encoder to each client; at least two clients process local business data based on the encoder parameters to obtain local business feature statistics, and send the local business feature statistics to the server; the server then aggregates the local business feature statistics of each client into global business feature statistics, determines the target pseudo feature based on the global business feature statistics and the initial pseudo feature, and adjusts the classifier in the business processing model according to the target pseudo feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object. The target pseudo feature is determined by the global business feature statistics and the initial pseudo feature, and the distribution of the initial pseudo feature is aligned to the distribution of the real global business feature, thereby improving the accuracy of the synthetic pseudo feature, and adjusting the classifier of the business processing model based on the determined target pseudo feature, that is, using the pseudo feature whose distribution is similar to the distribution of the real global business feature to train the classifier, the server does not need to obtain the real business feature of the client, thereby improving the classification accuracy of the business processing model while protecting privacy, and thus can improve the accuracy of classifying the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a scenario diagram of a federated long-tail learning method based on synthetic pseudo features provided by an embodiment of this specification; Figure 2 is a flowchart of a federated long-tail learning method based on synthetic pseudo features provided by an embodiment of this specification; Figure 3 is a schematic diagram of optimizing target pseudo features according to total loss values ​​provided by an embodiment of this specification; Figure 4 is a processing flow chart of a federated long-tail learning method based on synthetic pseudo features provided by an embodiment of this specification; Figure 5 is a structural diagram of a data processing system provided by an embodiment of this specification; Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0013] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0014] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0016] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0017] First, the terms involved in one or more embodiments of this specification are explained.

[0018] Federated long-tailed learning: Federated long-tailed learning refers to a method to solve the long-tail distribution problem of data under the federated learning framework.

[0019] Pseudo feature: Pseudo feature usually refers to those features that are not directly derived from the original data, but are generated or derived in some way.

[0020] RFF: RFF (Random Fourier Features) is a method for approximating kernel functions and is suitable for large-scale machine learning tasks.

[0021] Cholesky decomposition: Cholesky decomposition is a method of decomposing a symmetric positive definite matrix into the product of a lower triangular matrix and its transpose. The decomposed lower triangular matrix is ​​called a Cholesky factor.

[0022] MMD: MMD (Maximum Mean Discrepancy) is a statistic used to measure the difference between two probability distributions.

[0023] Non-IID: Non-IID (Non-Independent and Identically Distributed) means that the data samples are not independent of each other, or they do not come from the same probability distribution.

[0024] Secure Aggregation Protocol: Secure Aggregation Protocol is a protocol for distributed or multi-party data aggregation. Its main function is to protect the data privacy of individual participants during the aggregation process. It is usually used in scenarios where data needs to be aggregated among multiple participants but individual data does not want to be leaked, such as federated learning, privacy-preserving statistical analysis, etc.

[0025] ReLU activation function: ReLU (Rectified Linear Unit) is a very popular activation function, especially widely used in deep learning. The value range of the ReLU activation function is [0, +∞).

[0026] Sigmoid activation function: The sigmoid activation function is a classic activation function. The value range of the sigmoid activation function is (0, 1).

[0027] This specification proposes a federated long-tail learning method based on synthetic pseudo features, which synthesizes pseudo features by aligning low-order and high-order statistics, and uses the synthesized pseudo features to retrain the classifier, thereby correcting the classifier bias generated in long-tail learning. Moreover, under the condition of complying with the privacy protection requirements of federated learning, the learning effect of the model on long-tail data is effectively improved.

[0028] In this specification, a federated long-tail learning method based on synthetic pseudo features is provided. This specification also involves a data processing system, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0029] See also Figure 1 , Figure 1 A schematic diagram of a scenario of a federated long-tail learning method based on synthetic pseudo features provided according to an embodiment of the present specification is shown, specifically including: In response to the model adjustment request, the server splits the business processing model to obtain an encoder, and sends the encoder parameters of the encoder to each client, including client 1 to client k, where the value of k is determined according to the actual number of clients included in the data processing system; each client receives the encoder parameters, and based on the encoder parameters, calls the encoder of the business processing model in the server, extracts features of the local business data of each client to obtain business data features Z1, Z2...Zn; calculates local business feature statistics based on data features and preset statistical rules, including calculating the local feature mean , local characteristic second-order moment and the local RFF mean ; The local business feature statistics of each client are masked based on the security aggregation protocol and sent to the server; the server aggregates the local business feature statistics based on the security aggregation protocol to obtain global business feature statistics, including the global feature mean , global feature covariance , global RFF mean ; Obtain the initial pseudo features in the pseudo feature module, adjust and transform the initial pseudo features, align them with the global business feature distribution, and synthesize the target pseudo features; retrain the classifier of the business processing model based on the target pseudo features to obtain the adjusted target business processing model; the classification accuracy of the retrained target business processing model is improved.

[0030] In practical applications, there is data loss in the process of synthesizing pseudo features based on initial pseudo features and global business feature statistics, so the pseudo feature synthesis loss value can be optimized, specifically including: after synthesizing the above-mentioned target pseudo features, the pseudo feature module is updated using back propagation and gradient descent algorithms based on the pseudo feature synthesis loss value; then new initial pseudo features are sampled from the updated pseudo feature module to determine new target pseudo features, and the specific synthesis method is consistent with the steps of synthesizing pseudo features described above; the process of updating the pseudo feature module based on the pseudo feature synthesis loss value is iteratively performed until the preset number of iterations is reached; after the preset number of iterations is reached, the final initial pseudo features are sampled from the final pseudo feature module, and the target pseudo features are determined based on the initial pseudo features, and the classifier is retrained based on the current target pseudo features to obtain the final adjusted target business processing model. The target business processing model obtained through retraining is then used to classify the target object, thereby improving the accuracy of classifying the target object.

[0031] See also Figure 2 , Figure 2A flowchart of a federated long-tail learning method based on synthetic pseudo features provided according to an embodiment of the present specification is shown, which is applied to a data processing system, wherein the data processing system includes a server and at least two clients, and specifically includes the following steps: Step 202: The server splits the service processing model to obtain an encoder, and sends encoder parameters of the encoder to each client.

[0032] Among them, the server refers to the server in the data processing system for federated learning, which can cooperate with the clients in the data processing system to perform model training. The client refers to the client in the data processing system for federated learning, which can cooperate with the server in the data processing system to perform model training; the business processing model refers to the neural network model obtained by model training based on federated learning; the encoder refers to the partial model for feature extraction obtained by splitting the business processing model; the encoder parameters refer to the model parameters corresponding to the encoder. The client can call the encoder in the server by obtaining the encoder parameters, thereby realizing feature extraction on the client; the data processing system refers to a system for model training or model optimization based on federated learning, which includes a server and at least two clients. Model training or model optimization can be achieved through interaction between the server and the client in the system.

[0033] Specifically, the server receives an optimization request for the business processing model, splits the business processing model in response to the optimization request, obtains an encoder and a classifier, the encoder is used to extract features from the input data, and the classifier is used to classify the input features; obtains encoder parameters corresponding to the encoder, and sends the encoder parameters to each client participating in the training of the federated learning model, so that the client can call the server's encoder based on the encoder parameters to extract features.

[0034] In a specific implementation of the present specification, a data processing system based on federated learning includes a server A and clients B1 and B2; the data processing system obtains a trained business processing model through model training; server A receives an optimization request for the business processing model, splits the business processing model in response to the optimization request, and obtains an encoder; obtains encoder parameters corresponding to the encoder, and sends the encoder parameters to clients B1 and B2 respectively.

[0035] By sending the encoder parameters to each client, each client can subsequently call the encoder on the server based on the encoder parameters, and then extract local business features based on the encoder, aggregate the local business features into local business feature statistics, which serve as a standard for subsequent optimization of the business processing model.

[0036] Step 204: At least two clients process the local service data based on the encoder parameters to obtain local service feature statistics, and send the local service feature statistics to the server.

[0037] Among them, local business data refers to the business data stored locally on each client; local business feature statistics refer to business feature statistics obtained by extracting and performing feature extraction and statistical processing on the local business data based on the encoder.

[0038] Specifically, the local service data is processed based on the encoder parameter to obtain local service feature statistics, including: extracting local service features from the local service data based on the encoder parameters; The local service feature statistics are calculated according to the feature statistics rule and the local service feature.

[0039] Among them, local business features refer to sample features extracted by the encoder from local business data; feature statistical rules refer to rules for statistically analyzing the extracted business features, which may include mean statistical rules, second-order moment statistical rules, RFF mean statistical rules, and sample quantity statistical rules.

[0040] Specifically, for each data category c, each client performs statistics and calculations on the feature mean, feature second-order moment, feature RFF mean, and sample quantity of the local business data to obtain local business feature statistics. , local characteristic second-order moment , local RFF mean , Sample size For example, the following formulas (1)-(4) are used as mean statistical rules, second-order moment statistical rules, RFF mean statistical rules, and sample quantity statistical rules to perform statistics and calculations on local service feature statistics. In the following formulas, k represents the number of clients, c represents the feature category, Indicates that the business characteristic data , All the business features of the cth category extracted from , as shown in the following formula (5); for a certain sample feature z, the calculation method of its RFF is based on the following formula (6), where D represents the RFF dimension: Formula (1) Formula (2) Formula (3) Formula (4) Formula (5) Formula (6) The local business feature statistics obtained through statistics and calculation are sent to the server so that the local business feature statistics can be further processed later.

[0041] Step 206: The server aggregates the local business feature statistics of each client into a global business feature statistics, determines the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjusts the classifier in the business processing model according to the target pseudo-feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object.

[0042] Among them, the initial pseudo-feature refers to the unaligned pseudo-feature sampled from the pseudo-feature module; the target pseudo-feature refers to the pseudo-feature obtained by adjusting the initial pseudo-feature based on the global business feature statistics, and the distribution is close to the real global business feature distribution; the synthesized target pseudo-feature is input into the classifier as a sample feature to achieve fine-tuning of the classifier, so that the classification accuracy of the target business processing model is improved when classifying the target object; the target object refers to the object that has classification requirements in the target business scenario, for example, in the recruitment scenario, the resume of the applicant is classified, that is, the resume is the target object in this business scenario.

[0043] In the federated learning scenario, the data privacy of each client needs to be protected; therefore, after the server receives the local business feature statistics of each client, it is necessary to securely aggregate the local business feature statistics of each client to obtain the global business feature statistics, and then perform subsequent data processing to further ensure the security of the data.

[0044] Specifically, determining the target pseudo feature based on the local service feature statistics of each client and the initial pseudo feature of the service processing model includes: Sampling initial pseudo features in a pseudo feature module; Aggregate the local service feature statistics of each client according to the security aggregation protocol to obtain global service feature statistics; A target pseudo feature is determined based on the global service feature statistics and the initial pseudo feature.

[0045] The pseudo feature module may be in the form of a memory, which is used to store the initial pseudo features, and the stored initial pseudo features are randomly initialized; the pseudo feature module may also be in the form of a generator, which is a neural network for generating the initial pseudo features, and the model parameters of the neural network are randomly initialized; The global service feature statistics refer to the service feature statistics obtained by aggregating the local service feature statistics of each client; aggregating the local service feature statistics of each client refers to calculating the global service feature statistics respectively through the following formulas (7)-(9), including the global feature mean , global feature covariance , global RFF mean , in the formula Represents the total number of samples in category c: Formula (7) Formula (8) Formula (9) The above formulas (7)-(9) are all in the form of weighted average. It is in the form of summation. This form of aggregation operation supports aggregating the local business feature statistics of each client through a secure aggregation protocol, so that the server only obtains the global business feature statistics and cannot determine the local business features of a certain client, thereby protecting the privacy of each client.

[0046] Further, determining a target pseudo feature based on the global service feature statistics and the initial pseudo feature includes: Decomposing the global feature covariance in the global service feature statistics to obtain a global decomposition factor; Calculate the initial pseudo feature covariance and the initial pseudo feature mean based on the initial pseudo feature; Decomposing the initial pseudo feature covariance to obtain an initial decomposition factor; According to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global business feature statistics, the initial pseudo feature is converted to obtain a target pseudo feature.

[0047] Among them, the global business feature statistics may include the global feature mean and the global feature covariance; the global decomposition factor refers to the matrix obtained by decomposing the global feature covariance based on a preset decomposition algorithm; the initial pseudo-feature covariance refers to the covariance calculated based on the initial pseudo-feature, and the initial pseudo-feature mean refers to the mean calculated based on the initial pseudo-feature; the initial decomposition factor refers to the matrix obtained by decomposing the initial pseudo-feature covariance based on a preset decomposition algorithm.

[0048] Specifically, the global feature mean and global feature covariance in the global business feature statistics, as well as the initial pseudo feature covariance and initial pseudo feature mean of the initial pseudo feature are determined to adjust the initial pseudo feature; the initial pseudo feature distribution is adjusted to the distribution of the real global business feature, so as to align the pseudo feature with the real feature and realize the synthesis of the target pseudo feature. The global feature covariance and the initial pseudo feature covariance are decomposed by a preset decomposition algorithm, and the initial pseudo feature mean is calculated to determine the adjustment range of the initial pseudo feature.

[0049] In a specific embodiment of this specification, by Align and obtain the target pseudo features , the calculation is completed based on the following formulas (10)-(13): Formula (10) Formula (11) Formula (12) Formula (13) in, represents the global feature covariance, represents the global decomposition factor, represents the initial pseudo feature covariance, represents the initial decomposition factor, represents the linear transformation matrix, represents the initial pseudo feature mean, Represents the global feature mean. Formula (10) uses the Cholesky algorithm to decompose the global feature covariance, and formula (11) uses the Cholesky algorithm to decompose the initial pseudo feature covariance. is the first-order statistic, the global feature covariance is a second-order statistic, so the process of determining the target pseudo-feature achieves the alignment of low-order statistics.

[0050] It should be noted that since the Cholesky decomposition is only applicable to positive definite matrices, and for the covariance matrix, full rank is equivalent to positive definite, a full rank constraint is required to let The Cholesky decomposition of was successfully performed. In order to make its Cholesky decomposition successful, a positive number less than the preset threshold can be added to the diagonal ,Right now Due to the semi-positive definiteness of the covariance matrix, the diagonal plus It can be changed with a small change in value. Convert to a positive definite matrix.

[0051] The global feature covariance and the initial pseudo-feature covariance are decomposed by a preset decomposition algorithm. The initial pseudo-feature is adjusted based on the linear transformation matrix obtained based on the decomposition result to align the pseudo-feature distribution with the real global business feature distribution, and the aligned target pseudo-feature is obtained.

[0052] Furthermore, the target pseudo-features obtained based on the above method can be used to adjust the classifier to optimize the business processing model, but in order to make the target pseudo-features closer to the real features, that is, have a higher similarity with the real features, the optimized target pseudo-features can be obtained based on the following method.

[0053] Preferably, in another specific implementation manner of the present specification, determining a target pseudo feature based on the global service feature statistics and the initial pseudo feature includes: Decomposing the global feature covariance in the global service feature statistics to obtain a global decomposition factor; Calculate the initial pseudo feature covariance and the initial pseudo feature mean based on the initial pseudo feature; Decomposing the initial pseudo feature covariance to obtain an initial decomposition factor; According to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global service feature statistics, the initial pseudo feature is converted to obtain a current pseudo feature; A target pseudo feature is determined based on the current pseudo feature, the initial pseudo feature and a global Fourier feature mean in the global business feature statistics.

[0054] Among them, the current pseudo-feature refers to the pseudo-feature calculated based on the method described above, which will not be elaborated here; the global business feature statistics can also include the global Fourier feature mean, which is used to optimize the current pseudo-feature to obtain the target pseudo-feature.

[0055] Specifically, the loss value generated in the current pseudo-feature synthesis process is determined, and the pseudo-feature module of the business processing model is updated based on the loss value, and then the target pseudo-features are synthesized based on the pseudo-features in the updated pseudo-feature model, thereby improving the accuracy of the target pseudo-features, that is, making the target pseudo-features more similar to the real features.

[0056] Specifically, determining a target pseudo feature based on the current pseudo feature, the initial pseudo feature and a global Fourier feature mean in the global service feature statistics includes: Obtaining a constraint loss value of the initial pseudo feature; Calculating a maximum mean difference loss value based on the global Fourier feature mean; Regularizing the current pseudo feature to obtain a regularized loss value; A total loss value is determined according to the constraint loss value, the maximum mean difference loss value, and the regularization loss value, and a target pseudo feature is determined based on the total loss value.

[0057] Among them, the constraint loss value refers to the value generated when constraining the rank of the initial pseudo-feature. Since the rank of the matrix cannot be increased during the linear transformation, that is, the process of aligning the initial pseudo-feature with the global business feature, the rank of the initial pseudo-feature should be greater than or equal to the rank of the real feature. The full rank needs to be constrained by the decorrelation regularization term. For example, in this process, the following formula (14) is used to constrain the initial pseudo-feature, resulting in a constraint loss value : Formula (14) MMD can be used to calculate the difference between two data sets, thereby reducing the distribution difference between synthetic pseudo features and real global business features. However, in federated learning, there is a need to protect data privacy, and MMD cannot be calculated directly. Therefore, this manual uses RFF to calculate the approximate MMD. The specific calculation method is as follows: Formula (15) The above formula (15) is based on the data set using RFF and The approximate MMD is calculated by calculating the maximum difference value of is the mapping function of RFF, which is used to calculate the Gaussian kernel function: .

[0058] The approximate MMD loss value optimized when synthesizing features, that is, the maximum mean difference loss value Obtained by the following formula (16): Formula (16) in, Indicates the number of current pseudo features, represents the global Fourier eigenvalue mean, Represents the Fourier feature mean of the current pseudo feature. Since the Gaussian kernel is used in MMD, the minimization The high-order statistics of the synthetic features will be implicitly aligned with the statistics of the real features, so that the distribution of the synthetic features gradually approaches the distribution of the real features.

[0059] By optimizing the approximate MMD, implicit high-order alignment of features is achieved, further narrowing the difference between the distribution of synthesized target pseudo-features and the distribution of true global business features.

[0060] The value range of the real feature will be affected by the encoder activation function, so it is necessary to adjust the value range of the current pseudo feature by regularizing the value range of the current pseudo feature; taking the activation function of the encoder as the ReLU activation function as an example, the regularization loss value generated by regularization is Based on the following formula (17): Formula (17) Taking the sigmoid activation function as an example, the regularized loss value generated by regularization is Based on the following formula (18): Formula (18) By calculating the regularization loss value, the target pseudo-feature data range is regularized, making the target pseudo-feature data more accurate.

[0061] Based on the constraint loss value, the maximum mean difference loss value and the regularization loss value, the total loss of feature synthesis is further calculated, that is, the total loss value , see the following formula (19) for details: Formula (19) in, represents the weight of the regularization loss value, The weight representing the constraint loss value.

[0062] Further, after determining the total loss value, determining the target pseudo feature based on the total loss value includes: Update the pseudo feature module based on the total loss value to obtain a target pseudo feature module; Sampling the current initial pseudo feature in the target pseudo feature module; A target pseudo feature is determined based on the global service feature statistics and the current initial pseudo feature.

[0063] Specifically, based on the total loss value, the back propagation and gradient descent algorithms are used to update the pseudo feature module, and the current initial pseudo features are sampled in the updated target pseudo feature module.

[0064] Based on the total loss value, the pseudo feature module is updated by back propagation and gradient descent algorithm. If the pseudo feature module is in the form of a memory, the pseudo feature module is updated by directly updating the initial pseudo feature stored in the memory; if the pseudo feature module is in the form of a generator, the pseudo feature module is updated by updating the model parameters of the generator neural network. The current initial pseudo feature is sampled in the updated final pseudo feature module.

[0065] Further, determining a target pseudo feature based on the global service feature and the current initial pseudo feature includes: Calculate the current initial pseudo feature covariance and the current initial pseudo feature mean based on the current initial pseudo feature; Decomposing the current initial pseudo feature covariance to obtain a current decomposition factor; According to the global decomposition factor, the current decomposition factor, the current initial pseudo feature mean and the global feature mean, the current initial pseudo feature is converted to obtain a target pseudo feature.

[0066] Among them, the current initial pseudo-feature covariance refers to the covariance calculated based on the current initial pseudo-feature; the current initial pseudo-feature mean refers to the mean calculated based on the current initial pseudo-feature; and the current decomposition factor refers to the matrix obtained by decomposing the current initial pseudo-feature covariance based on a preset decomposition algorithm.

[0067] Specifically, the global feature mean and global feature covariance in the global business feature statistics, as well as the current initial pseudo feature covariance and the current initial pseudo feature mean of the current initial pseudo feature are determined to adjust the current initial pseudo feature; the current initial pseudo feature distribution is adjusted to the distribution of the global business feature, so as to align the pseudo feature with the real feature and realize the synthesis of the target pseudo feature. The global feature covariance and the current initial pseudo feature covariance are decomposed by a preset decomposition algorithm, and the current initial pseudo feature mean is calculated to determine the adjustment range of the current initial pseudo feature.

[0068] In a specific embodiment of this specification, Figure 3 As shown in a schematic diagram of optimizing the target pseudo-feature according to the total loss value, the current initial pseudo-feature sampled from the target pseudo-feature module can be aligned with the global business feature statistics by referring to the above formulas (10)-(13) to obtain the target pseudo-feature; the target pseudo-feature can be iteratively optimized, so the target pseudo-feature can be further optimized based on the above formulas (14)-(19) until the preset number of iterations is reached, and then the classifier is retrained based on the optimized target pseudo-feature. Since the specific calculation method is the same as above, it will not be repeated here.

[0069] The obtained target pseudo features and the corresponding category labels are constructed into a pseudo feature dataset, and the classifier of the business processing model is retrained based on the target pseudo features in the pseudo feature dataset, so as to correct the deviation of the classifier in the business processing model and obtain a target business processing model with higher classification accuracy for business data.

[0070] An embodiment of the present specification realizes that a server splits a business processing model to obtain an encoder, and sends the encoder parameters of the encoder to each client; at least two clients process local business data based on the encoder parameters to obtain local business feature statistics, and send the local business feature statistics to the server; the server then synthesizes a target pseudo feature based on the global feature statistics and the initial pseudo feature of the business processing model, and adjusts the classifier in the business processing model according to the target pseudo feature to obtain a target business processing model, wherein the target business processing model is used to classify the target object. Pseudo features are synthesized by synthesizing pseudo features from global feature statistics and initial pseudo features, and the distribution of pseudo features is aligned to the distribution of real global business features, thereby improving the accuracy of synthesized pseudo features. The classifier of the business processing model is adjusted based on the synthesized pseudo features, that is, the classifier is trained using pseudo features whose distribution is similar to the distribution of real global business features. The server does not need to obtain the real business features of the client, thereby improving the classification accuracy of the business processing model while protecting privacy, and thus improving the accuracy of classifying the target object.

[0071] The following combination Figure 4 Taking the application of the federated long-tail learning method based on synthetic pseudo features provided in this specification in a medical scenario as an example, the federated long-tail learning method based on synthetic pseudo features is further described. Figure 4 A flowchart of a processing process of a federated long-tail learning method based on synthetic pseudo features provided by an embodiment of this specification is shown, which specifically includes the following steps: Step 402: The server in the medical data processing system splits the disease classification model in response to the model optimization request, and sends encoder parameters of the encoder in the disease classification model to at least two clients.

[0072] Specifically, before the medical data processing system receives the model optimization request, it responds to the model training request and trains the disease classification model in a federated learning training manner. The encoder in the disease classification model is separated and the encoder parameters of the encoder are sent to each client in the medical data processing system.

[0073] Step 404: Each client processes the local service data based on the encoder parameters to obtain local service feature statistics, and returns the local service feature statistics to the server.

[0074] Specifically, the encoder is called based on the encoder parameters to extract local business features from the local business data; and local business feature statistics are calculated according to feature statistical rules and local business features.

[0075] Step 406: The server aggregates the local business feature statistics of each client into a global business feature statistics, determines the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjusts the classifier in the disease classification model according to the target pseudo-feature to obtain a target disease classification model, wherein the target disease classification model is used to classify the patient's disease information.

[0076] Specifically, the local business feature statistics of each client are aggregated based on a secure aggregation protocol to obtain global business feature statistics; initial pseudo features are sampled in the pseudo feature module of the disease classification model; the global feature covariance in the global business feature statistics is decomposed to obtain a global decomposition factor; the initial pseudo feature covariance and the initial pseudo feature mean are calculated based on the initial pseudo features; the initial pseudo feature covariance is decomposed to obtain an initial decomposition factor; the initial pseudo feature is converted according to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global business feature statistics to obtain a target pseudo feature.

[0077] Furthermore, the total loss value generated during the synthesis of the target pseudo-features is calculated; the pseudo-feature module of the disease classification model is updated based on the total loss value so as to obtain new initial pseudo-features in the updated pseudo-feature module; the above process of obtaining the initial pseudo-features in the pseudo-feature module, determining the target pseudo-features based on the initial pseudo-features, and calculating the total loss value for updating the pseudo-feature module is repeated until a preset number of iterations is reached to obtain the final target pseudo-features; a pseudo-feature dataset is constructed based on the final target pseudo-features, and the classifier of the disease classification model is retrained to obtain the target disease classification model.

[0078] An embodiment of the present specification realizes that a server in a medical data processing system responds to a model optimization request, splits a disease classification model, and sends encoder parameters of an encoder in the disease classification model to at least two clients; each client extracts local business features from local business data based on the encoder parameters, and calculates local business feature statistics based on the local business features and returns them to the server; the server aggregates the local business feature statistics of each client into global business feature statistics, and determines target pseudo features based on the global business feature statistics and initial pseudo features, and adjusts the classifier in the disease classification model according to the target pseudo features to obtain a target disease classification model, wherein the target disease classification model is used to classify the patient's disease information. Pseudo features are synthesized by global feature statistics and initial pseudo features, and the distribution of pseudo features is aligned to the distribution of real global business features, thereby improving the accuracy of the synthesized target pseudo features. The classifier of the disease classification model is adjusted based on the synthesized pseudo features, that is, the classifier is trained using pseudo features whose distribution is similar to the distribution of real global business features, and the server does not need to obtain the real business features of the client, thereby improving the classification accuracy of the disease classification model while protecting privacy, and further improving the classification accuracy of disease information.

[0079] Corresponding to the above method embodiment, this specification also provides a data processing system embodiment. Figure 5 FIG. 1 shows a schematic diagram of a data processing system provided by an embodiment of the present specification. Figure 5 As shown, the data processing system includes a server and at least two clients, wherein: The server 502 is configured to split the service processing model to obtain an encoder, and send encoder parameters of the encoder to each client; The at least two clients 504 are configured to process the local service data based on the encoder parameters to obtain local service feature statistics, and send the local service feature statistics to the server; The server 502 is configured to aggregate the local business feature statistics of each client into a global business feature statistics, determine the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjust the classifier in the business processing model according to the target pseudo-feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object.

[0080] The at least two clients 504 are further configured as follows: extracting local service features from the local service data based on the encoder parameters; The local service feature statistics are calculated according to the feature statistics rule and the local service feature.

[0081] The server 502 is further configured to: Sampling initial pseudo features in a pseudo feature module; Aggregate the local service feature statistics of each client according to the security aggregation protocol to obtain global service feature statistics; A target pseudo feature is determined based on the global service feature statistics and the initial pseudo feature.

[0082] The server 502 is further configured to: decompose the global feature covariance in the global service feature statistics to obtain a global decomposition factor; Calculate the initial pseudo feature covariance and the initial pseudo feature mean based on the initial pseudo feature; Decomposing the initial pseudo feature covariance to obtain an initial decomposition factor; According to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global business feature statistics, the initial pseudo feature is converted to obtain a target pseudo feature.

[0083] The server 502 is further configured to: decompose the global feature covariance in the global service feature statistics to obtain a global decomposition factor; Calculate the initial pseudo feature covariance and the initial pseudo feature mean based on the initial pseudo feature; Decomposing the initial pseudo feature covariance to obtain an initial decomposition factor; According to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global service feature statistics, the initial pseudo feature is converted to obtain a current pseudo feature; A target pseudo feature is determined based on the current pseudo feature, the initial pseudo feature and a global Fourier feature mean in the global business feature statistics.

[0084] The server 502 is further configured to: Obtaining a constraint loss value of the initial pseudo feature; Calculating a maximum mean difference loss value based on the global Fourier feature mean; Regularizing the current pseudo feature to obtain a regularized loss value; A total loss value is determined according to the constraint loss value, the maximum mean difference loss value, and the regularization loss value, and a target pseudo feature is determined based on the total loss value.

[0085] The server 502 is further configured to: Update the pseudo feature module based on the total loss value to obtain a target pseudo feature module; Sampling the current initial pseudo feature in the target pseudo feature module; A target pseudo feature is determined based on the global service feature statistics and the current initial pseudo feature.

[0086] The server 502 is further configured to: Calculate the current initial pseudo feature covariance and the current initial pseudo feature mean based on the current initial pseudo feature; Decomposing the current initial pseudo feature covariance to obtain a current decomposition factor; According to the global decomposition factor, the current decomposition factor, the current initial pseudo feature mean and the global feature mean, the current initial pseudo feature is converted to obtain a target pseudo feature.

[0087] The data processing system of the present specification realizes that the server splits the business processing model to obtain an encoder, and sends the encoder parameters of the encoder to each client; at least two clients process local business data based on the encoder parameters to obtain local business feature statistics, and send the local business feature statistics to the server; the server then aggregates the local business feature statistics of each client into global business feature statistics, synthesizes target pseudo features based on the global business feature statistics and the initial pseudo features sampled from the pseudo feature module, and adjusts the classifier in the business processing model according to the target pseudo features to obtain a target business processing model, wherein the target business processing model is used to classify the target object. Pseudo features are synthesized by synthesizing pseudo features from global business feature statistics and initial pseudo features, and the distribution of pseudo features is aligned to the distribution of real global business features, thereby improving the accuracy of synthesized pseudo features. The classifier of the business processing model is adjusted based on the synthesized pseudo features, that is, the classifier is trained using pseudo features whose distribution is similar to the distribution of real global business features, thereby improving the classification accuracy of the business processing model while protecting privacy, and further improving the accuracy of classifying the target object.

[0088] The above is a schematic scheme of a data processing system of this embodiment. It should be noted that the technical scheme of the data processing system and the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features belong to the same concept, and the details not described in detail in the technical scheme of the data processing system can be referred to the description of the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features.

[0089] Figure 6 The structure block diagram of a computing device 600 provided according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0090] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0091] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0092] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0093] Among them, when the processor 620 executes the computer program / instructions, the steps of the federated long-tail learning method based on synthetic pseudo features are implemented.

[0094] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features.

[0095] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned federated long-tail learning method based on synthetic pseudo-features.

[0096] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features.

[0097] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned federated long-tail learning method based on synthetic pseudo-features.

[0098] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the above-mentioned federated long-tail learning method based on synthetic pseudo features.

[0099] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0100] The computer program / instruction includes a computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0101] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0102] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0103] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A federated long-tail learning method based on synthetic pseudo features, applied to a data processing system, characterized in that: The data processing system comprises a server and at least two clients, wherein: The server splits the service processing model to obtain an encoder, and sends encoder parameters of the encoder to each client; The at least two clients process the local service data based on the encoder parameters to obtain local service feature statistics, and send the local service feature statistics to the server; The server aggregates the local business feature statistics of each client into global business feature statistics, determines the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjusts the classifier in the business processing model according to the target pseudo-feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object.

2. The method according to claim 1, characterized in that Processing the local service data based on the encoder parameters to obtain local service feature statistics includes: extracting local service features from the local service data based on the encoder parameters; The local service feature statistics are calculated according to the feature statistics rule and the local service feature.

3. The method according to claim 1, characterized in that Aggregating local service feature statistics of each client into global service feature statistics, and determining a target pseudo feature based on the global service feature statistics and the initial pseudo feature, including: Sampling initial pseudo features in the pseudo feature module; Aggregate the local service feature statistics of each client according to the security aggregation protocol to obtain global service feature statistics; A target pseudo feature is determined based on the global service feature statistics and the initial pseudo feature.

4. The method according to claim 3, characterized in that Determining a target pseudo feature based on the global service feature statistics and the initial pseudo feature includes: Decomposing the global feature covariance in the global service feature statistics to obtain a global decomposition factor; Calculate the initial pseudo feature covariance and the initial pseudo feature mean based on the initial pseudo feature; Decomposing the initial pseudo feature covariance to obtain an initial decomposition factor; According to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global business feature statistics, the initial pseudo feature is converted to obtain a target pseudo feature.

5. The method according to claim 3, characterized in that Determining a target pseudo feature based on the global service feature statistics and the initial pseudo feature includes: Decomposing the global feature covariance in the global service feature statistics to obtain a global decomposition factor; Calculate the initial pseudo feature covariance and the initial pseudo feature mean based on the initial pseudo feature; Decomposing the initial pseudo feature covariance to obtain an initial decomposition factor; According to the global decomposition factor, the initial decomposition factor, the initial pseudo feature mean and the global feature mean in the global service feature statistics, the initial pseudo feature is converted to obtain a current pseudo feature; A target pseudo feature is determined based on the current pseudo feature, the initial pseudo feature and a global Fourier feature mean in the global business feature statistics.

6. The method according to claim 5, characterized in that Determining a target pseudo feature based on the current pseudo feature, the initial pseudo feature and a global Fourier feature mean in the global service feature statistic includes: Obtaining a constraint loss value of the initial pseudo feature; Calculate a maximum mean difference loss value based on the global Fourier feature mean; Regularizing the current pseudo feature to obtain a regularized loss value; A total loss value is determined according to the constraint loss value, the maximum mean difference loss value, and the regularization loss value, and a target pseudo feature is determined based on the total loss value.

7. The method according to claim 6, characterized in that Determining a target pseudo feature based on the total loss value includes: Update the pseudo feature module based on the total loss value to obtain a target pseudo feature module; Sampling the current initial pseudo feature in the target pseudo feature module; A target pseudo feature is determined based on the global service feature statistics and the current initial pseudo feature.

8. The method according to claim 7, characterized in that Determining a target pseudo feature based on the global service feature statistics and the current initial pseudo feature includes: Calculate the current initial pseudo feature covariance and the current initial pseudo feature mean based on the current initial pseudo feature; Decomposing the current initial pseudo feature covariance to obtain a current decomposition factor; According to the global decomposition factor, the current decomposition factor, the current initial pseudo feature mean and the global feature mean, the current initial pseudo feature is converted to obtain a target pseudo feature.

9. A data processing system, characterized in that: The data processing system comprises a server and at least two clients, wherein: The server is configured to split the service processing model to obtain an encoder, and send encoder parameters of the encoder to each client; The at least two clients are configured to process the local service data based on the encoder parameters to obtain local service feature statistics, and send the local service feature statistics to the server; The server is configured to aggregate the local business feature statistics of each client into a global business feature statistics, determine the target pseudo-feature based on the global business feature statistics and the initial pseudo-feature, and adjust the classifier in the business processing model according to the target pseudo-feature to obtain the target business processing model, wherein the target business processing model is used to classify the target object.

10. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. It is characterized in that when the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing a computer program / instruction, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images