Asynchronous federation continuous learning method for large model

Abstract

The invention discloses an asynchronous federated continuous learning method for a large model, which belongs to the field of federated learning and continuous learning, is applied to a client, and comprises the following steps: receiving an initial model parameter of a global model issued by a server and a global feature extraction module, and performing feature representation on local service task data based on the module, constructing a corresponding feature prototype and a feature covariance statistic; uploading the feature prototype and the feature covariance statistics to a server in an asynchronous mode to perform asynchronous federal fusion, and updating global feature statistics; and receiving the global feature statistics after the server fuses and updates, wherein the global feature statistics are used for local decision modeling of subsequent tasks. Through the asynchronous federal aggregation strategy, incremental feature knowledge from different financial institutions and different task stages can be continuously integrated, the problem of historical knowledge forgetting is effectively relieved, and the stability and consistency of global feature representation in the long term evolution process are kept.

Description

Technical Field

[0001] This invention relates to the field of federated learning and continuous learning, and in particular to an asynchronous federated continuous learning method for large models. Background Technology

[0002] With the continuous improvement of the digitalization and intelligence of financial services, large-scale models have been widely used in core financial scenarios such as risk identification, anti-money laundering, credit assessment, and customer profiling. To fully utilize dispersed data resources among multiple financial institutions and meet the security requirement that financial data cannot leave the domain, federated learning has gradually become an important technical means for cross-institutional model collaborative training. Federated Learning (FL) is a distributed optimization framework for collaborative modeling among multiple participants. Its goal is to achieve joint optimization of the global model through parameter or statistical interaction without centralizing the original data. A federated learning system consists of a central server node and M financial institution clients. Each client accesses its private financial dataset locally and obtains model update information through local optimization. The server is responsible for fusing updates from different clients to approximate the ideal model trained under the joint data distribution. Existing federated learning methods mostly adopt synchronous update mechanisms, requiring each participating institution to simultaneously complete local model training and upload parameters within a fixed training round. This synchronous mechanism faces many limitations in the actual financial environment, such as significant differences in computing power among different institutions, different frequencies of business data generation, and inconsistent model training trigger times, which can easily lead to training process blockage and reduce overall efficiency. Furthermore, financial business scenarios are characterized by continuous evolution and dynamic task arrival, with data distribution and business priorities changing constantly across different institutions over time. Traditional federated learning methods are typically designed for static tasks and lack effective support for continuous learning scenarios. When faced with new business tasks or changes in data distribution, they are prone to problems such as insufficient model adaptability or forgetting of historical knowledge. Therefore, it is necessary to propose a federated learning method that can support asynchronous collaboration and has continuous update capabilities to adapt to the heterogeneity, dynamism, and long-term evolution requirements in the training of large-scale financial models across multiple institutions. Summary of the Invention

[0003] The purpose of this invention is to overcome the problems of the prior art and provide an asynchronous federated continuous learning method for large models.

[0004] The objective of this invention is achieved through the following technical solution: an asynchronous federated continuous learning method for large models, with the client acting as the execution entity, the method comprising the following steps: The system receives the initial model parameters of the global model and the global feature extraction module from the server. The initial model parameters of the global model are obtained by the server based on public datasets or historical business data. The global feature extraction module is determined based on the initial model parameters of the large model, and the parameters of the global feature extraction module are in a frozen state. When processing local financial business tasks, the local business task data is represented by features based on the global feature extraction module, and corresponding feature prototypes and feature covariance statistics are constructed for different business categories; the feature prototype is the mean vector of the sample features of different business categories; the feature covariance statistics is the covariance matrix that reflects the distribution of sample features of different business categories. Incremental feature information, including feature prototypes and feature covariance statistics, is uploaded to the server asynchronously, enabling the server to perform asynchronous federated fusion of the incremental feature information and update the global feature statistics; the global feature statistics include global feature prototypes and global covariance matrix. The server receives updated global feature statistics, which are used for local decision modeling in subsequent tasks.

[0005] In one example, after representing the local business data based on the global feature extraction module, the following steps are also included: The extracted feature representation is calibrated to make the calibrated feature representation closer to a Gaussian distribution.

[0006] In one example, corresponding feature prototypes and feature covariance statistics are constructed for different business categories, including: Calculate the mean of the feature vectors of samples from each business category as the feature prototype; The feature prototypes constructed for the current task are weighted and fused with the historical feature prototypes according to the sample size ratio to obtain the fused and updated feature prototypes. The covariance matrix is ​​calculated based on the feature vectors of samples from each business category, and used as the feature covariance statistic. The feature covariance statistics constructed for the current task are weighted and fused with the historical feature covariance statistics according to the sample size ratio to obtain the fused and updated feature covariance statistics.

[0007] In one example, after constructing corresponding feature prototypes and feature covariance statistics for different business categories, the following is also included: The quality and contribution of local data on the client side are quantitatively evaluated to generate a client quality score; Incremental feature information, including feature prototypes, feature covariance statistics, and client quality scores, is uploaded to the server asynchronously. The server then detects differences in client data distribution based on the currently stored global feature prototypes, constructs adaptive fusion weights by combining them with the client quality scores, and then performs asynchronous federated fusion of feature prototypes and feature covariance statistics based on the adaptive fusion weights to update the global feature statistics.

[0008] In one example, the quantitative evaluation of the quality and contribution of local client data to generate a client quality score includes: Calculate the consistency index between the current task data and the historical data distribution; Calculate the class coverage metric of the client in the global label space during the current task; Calculate data noise index based on covariance matrix trace; The consistency index, category coverage index, and data noise index are weighted and summed to generate a client quality score.

[0009] In one example, the consistency metric is calculated based on the Euclidean distance between the mean of the current task features and the mean of the historical fused features; the category coverage metric is calculated based on the ratio of the size of the label set involved in the current task to the size of the global label set; and the data noise metric is calculated based on the mean of the trace of the covariance matrix of each category.

[0010] In one example, after the receiving server merges and updates the global feature statistics, it also includes: Construct a covariance-aware Bayesian classifier based on global feature prototypes and the global covariance matrix; Extract the feature representation of newly arrived business task data, calculate the Mahalanobis distance between the feature representation and the global feature prototype of each category based on the covariance-aware Bayesian classifier, and classify the corresponding business task data into the category with the smallest Mahalanobis distance.

[0011] It should be further noted that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0012] This invention also includes an asynchronous federated continuous learning method for large models, with a server as the execution entity, comprising the following steps: The initial model parameters of the global model are obtained by training on public datasets or historical business data. The global feature extraction module is determined, and its parameters are frozen. The initial model parameters and the global feature extraction module are distributed to each client, enabling the client to represent the local business task data based on the global feature extraction module when processing local financial business tasks. Corresponding feature prototypes and feature covariance statistics are constructed for different business categories. The client also asynchronously uploads incremental feature information, including feature prototypes and feature covariance statistics, to the server. The feature prototype is the mean vector of sample features for different business categories, and the feature covariance statistics are the covariance matrix reflecting the distribution of sample features for different business categories. Asynchronously receive incremental feature information from different clients; Incremental feature information is asynchronously federated and fused to update global feature statistics; the global feature statistics include global feature prototypes and global covariance matrix. The updated global feature statistics are distributed to each client for local decision-making modeling in subsequent client tasks.

[0013] In one example, after asynchronously receiving incremental feature information from different clients, the method further includes detecting differences in client data distribution based on the currently stored global feature prototype: Calculate the distribution offset of the corresponding category based on the feature prototype uploaded by the client and the global feature prototype currently stored on the server; Calculate the mean of the distribution offset of all business categories involved in the client, and use it as the overall distribution offset of the client; When the overall distribution offset exceeds a preset threshold, it is determined that the client has a significant non-independent identical distribution offset, and the client's fusion weight is reduced in asynchronous federated fusion.

[0014] In one example, the asynchronous federated fusion of incremental feature information to update global feature statistics further includes: Asynchronously receive incremental feature information from different clients; the incremental feature information includes feature prototypes, feature covariance statistics, and client quality scores, which are obtained by quantitatively evaluating the quality and contribution of the client's local data; The differences in client data distribution are detected based on the currently stored global feature prototypes. An adaptive fusion weight is constructed by combining the client quality score. Then, based on the adaptive fusion weight, the feature prototypes, feature covariance statistics, and global feature statistics are asynchronously federated and updated.

[0015] It should be further noted that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. The client constructs feature prototypes and feature covariance statistics through the global feature extraction module with frozen parameters, uploads incremental feature information asynchronously, and receives the fused and updated global feature statistics. This can eliminate training blockage caused by differences in computing power and business frequency of institutions in synchronous federated learning, improve the efficiency of cross-institutional collaboration, and continuously absorb new business knowledge, effectively suppress the forgetting of historical knowledge, thereby ensuring the consistency of global feature representation in long-term evolution.

[0017] Meanwhile, with the help of the above-mentioned asynchronous federated aggregation strategy, the server can continuously integrate incremental feature knowledge from different financial institutions and different task stages when client tasks arrive asynchronously and the task order is inconsistent. This effectively alleviates the problem of historical knowledge forgetting and maintains the stability and consistency of global feature representation in the long-term evolution process.

[0018] 2. By quantifying the quality and contribution of local client data and generating a client quality score, the server can combine the quality score to detect differences in client data distribution and construct adaptive fusion weights. This improves the asynchronous federated fusion effect, avoids the negative impact of low-quality or severely skewed clients on the global model, and enhances robustness in non-independent and identically distributed data environments.

[0019] 3. A covariance-aware Bayesian classifier is constructed based on global feature prototypes and global covariance matrix. Local decision modeling is performed through Mahalanobis distance, which makes full use of the correlation information between feature dimensions contained in the covariance matrix. It can accurately depict the true distribution shape of each category in the feature space, significantly improving the classification accuracy of new task data, especially when the sample size is limited or the category distribution is heterogeneous.

[0020] 4. By calculating the distribution offset between the feature prototype uploaded by the client and the global feature prototype stored on the server, it is possible to identify clients with significant non-independent identically distributed offsets and reduce their fusion weight in asynchronous federated fusion. This effectively suppresses the interference of abnormal clients or clients with severe distribution offsets on the global model, prevents the model from being biased by a few offset data, and ensures the stability of the global feature representation. Attached Figure Description

[0021] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0022] Figure 1 This is a flowchart of a method applied to a client according to an embodiment of the present invention; Figure 2 This is a flowchart of a method applied to a server according to an embodiment of the present invention; Figure 3 A flowchart of a preferred embodiment of the method provided by the present invention; Figure 4 A flowchart of another preferred embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] This invention's method can be applied to financial scenarios, healthcare scenarios, cybersecurity and anti-fraud scenarios, etc. The embodiments section uses the application of this method to a financial scenario as an example, where this invention provides an asynchronous federated continuous learning method for large financial models. A large financial model refers to a deep learning model trained on large-scale parameter structures and multi-source financial data, such as Transformer architecture and its variants, Long Short-Term Memory (LSTM) networks, and graph convolutional neural networks. These models typically possess strong feature representation capabilities and cross-task generalization abilities, and can be applied to various financial business scenarios such as risk control, anti-fraud, anti-money laundering, credit assessment, and transaction monitoring. Large financial models often have large parameter scales, high training costs, and are sensitive to changes in data distribution. Therefore, in the process of multi-institutional collaborative training and long-term business evolution, a reasonable distributed training mechanism and continuous learning strategy are required. This invention addresses the collaborative training needs of large financial models by proposing a holistic method adapted to asynchronous federation and continuous learning characteristics.

[0026] Furthermore, Continual Learning (CL) refers to a model's ability to continuously learn new knowledge while maintaining performance on previously learned tasks and minimizing or preventing catastrophic forgetting when faced with a series of tasks or data streams arriving in chronological order. In financial scenarios, due to the high sensitivity of inter-institutional data and strict data compliance constraints, federated learning has become a core technological path for achieving cross-institutional model collaboration. Federated Continual Learning refers to introducing a continuous learning mechanism within the federated learning framework, enabling the model to continuously update itself as tasks arrive from multiple clients and at multiple stages, while maintaining effective recognition of previously learned tasks. In this scenario, each client independently receives tasks based on its own business triggering rhythm. The arrival time and order of tasks vary across different clients, and each client can only use the local data corresponding to the current task for training at any given stage. During federated continuous learning, the global model needs to achieve the following objectives without sharing the original data: first, continuously integrate new task knowledge from different clients; second, avoid significant degradation of historical task performance caused by new task training; and third, adapt to the impact of inconsistent task order and data distribution differences between clients. In financial operations, new products, rules, and risk types constantly emerge. Models need to continuously absorb new business knowledge without significantly reducing their ability to identify historical business patterns due to training on new tasks. The method proposed in this invention belongs to the task-sequence-oriented asynchronous federated continuous learning approach. By constructing covariance knowledge statistics independent of task order and combining them with model update constraints for continuous learning, the server can stably update the global model while asynchronously receiving incremental information from the client model. This effectively reduces the risks of model drift and historical knowledge forgetting, improving the stability and adaptability of the federated model in the long-term evolution of financial operations.

[0027] Furthermore, the asynchronous federated update mechanism allows clients to upload model update information to the server immediately after completing local training, without waiting for other clients to enter the same training phase. The server can then integrate the updates received at any point in time and use them for the evolution of the global model. Compared with synchronous federated learning, the asynchronous mechanism has significant advantages in the following aspects: (1) avoiding slowing down the overall training progress due to limited computing power or low-frequency business operations of some institutions; (2) adapting to asynchronous task arrival, allowing different financial institutions to access new business tasks at different times and independently trigger model updates; (3) improving system throughput, with the server continuously absorbing incremental model information to achieve streaming model optimization. This invention does not rely on the consistency of client task order during the asynchronous federated update process, supporting long-term collaborative training across multiple institutions and stages at the system level.

[0028] In this invention, an Order-Asynchronous Federated Continual Learning (OAFCL) paradigm is introduced to address scenarios where the arrival order of tasks from multiple institutions is inconsistent in financial transactions. Under the OAFCL paradigm, each client independently receives and processes its own task sequence, while the global model aggregates knowledge from different clients on the server side to adapt to new tasks and maintain effective recognition of learned knowledge.

[0029] Assume that the asynchronous federated continuous learning system corresponding to the method of this invention contains M financial institution clients. Having a private dataset Dataset The dataset contains image and / or text data, including account information and transaction information. Account information includes account type, account opening date, account balance, and transaction frequency. Transaction information includes transaction amount, transaction time, and transaction type. Size varies depending on the client. Each client The business task sequence is ,in This indicates the total number of tasks performed by the client. Task types include credit assessment tasks, anti-fraud detection tasks, and risk warning tasks. (Client) The Task training set Represented as: ; in, Representation Class of A sample set; This indicates the input sample data, such as account information, transaction information, etc. Indicates category labels, such as fraudulent transactions, risky transactions, etc.; Indicate category The set of all input sample data; Indicates client The The set of category labels involved in each task. The corresponding test set. for: ; in, Indicate category The set of all test input samples; Indicates client The The set of test category labels involved in each task.

[0030] In the OAFCL scenario, all clients participate in the same classification task, but the subset of tags encountered by each client at the same point in time may be different, that is: ; in For the global tag set; and These represent the phases of clients a and b, respectively. The subset of tags encountered. For the same client, the tag sets may overlap between different tasks: ; in, Indicates client The The set of training business categories involved in each task; Indicates client The The set of training business categories involved in each task; , For tags.

[0031] Samples of the same category are not shared between different clients: ; in, Indicates category label 1; Indicates category label 2; Indicates client The training data belongs to the category The set of all samples; Indicates client The training data belongs to the category The set of all samples.

[0032] Client completes task Then, performance evaluation can be performed on the test samples of all completed tasks: .

[0033] in, Indicates client The test dataset for the first task; Indicates client No. Test dataset for each task.

[0034] Based on this, such as Figure 1 As shown, an asynchronous federated continuous learning method for large models is applied to the client side. The method includes the following steps: S1: Receive the initial model parameters of the global model and the global feature extraction module sent by the server; the initial model parameters of the global model are obtained by the server based on public datasets or historical business data. The global feature extraction module is determined based on the initial model parameters of the large model, and the parameters of the global feature extraction module are in a frozen state.

[0035] In step S1, the public dataset can be publicly available historical transaction data or open-source datasets in the financial field. Historical business data includes account information, transaction information, etc. Account information includes account type, account opening time, account balance, and transaction frequency. Transaction information includes transaction amount, transaction time, and transaction type. When the method of this invention is applied to other scenarios, only the corresponding dataset needs to be replaced. For example, when applied to the medical and health scenario, the public dataset or historical business data is medical imaging data or electronic medical records. Medical imaging data includes CT image sequences, MRI scan images, etc. When applied to the network security and anti-fraud scenario, the public dataset or historical business data is network traffic data, attack event data, etc. Network traffic data includes packet load characteristics, traffic statistics characteristics, protocol type, source and destination IP addresses and port numbers, communication duration, packet size distribution, etc. Attack event data includes intrusion detection alarm records, malware signatures, attack timestamps, attack type tags, threat intelligence indicators, vulnerability exploitation code characteristics, attack chain association information, etc.

[0036] Furthermore, the initial model parameters include model weights, biases, etc. Freezing the parameters of the global feature extraction module ensures consistency in the feature space. Preferably, each client retains a local copy of the model for generating task data feature representations.

[0037] S2: When processing local financial business tasks, the local business task data is represented by features based on the global feature extraction module, and corresponding feature prototypes and feature covariance statistics are constructed for different business categories to characterize the incremental knowledge acquired by the client in the current task stage.

[0038] Specifically, the client It receives the financial business task at its current time t Based on the global feature extraction module fixed in step S1 Feature representation of task data Furthermore, feature statistics reflecting distribution information and prototypes reflecting decision information are constructed, namely: the feature prototype is the mean vector of the features of samples from different business categories, and the feature covariance statistics is the covariance matrix reflecting the distribution of features of samples from different business categories.

[0039] S3: Upload incremental feature information, including feature prototypes and feature covariance statistics, to the server in an asynchronous manner, so that the server can perform asynchronous federated fusion of incremental feature information and update global feature statistics; the global feature statistics include global feature prototypes and global covariance matrix.

[0040] In step S3, the client terminals of each financial institution After completing local task processing and building continuous feature statistics for the current task phase, the incremental feature information is uploaded to the server asynchronously to participate in the subsequent federated aggregation process. Specifically, the client... The persistent feature statistics obtained based on step S2 at the current task stage t include: the fused feature covariance matrix. fusion of characteristic prototypes And the corresponding number of samples required.

[0041] S4: Receive the global feature statistics updated by the server, which are used for local decision modeling in subsequent tasks.

[0042] In step S4, the global feature statistics include the central location of each category (represented by the global feature prototype) and its distribution state (represented by the global covariance matrix). The client can classify and predict newly arriving business task data based on the global feature statistics and output the classification prediction results, such as classifying a transaction as a normal transaction or a fraudulent transaction. Similarly, when the method of this invention is applied to network security and anti-fraud scenarios, it can output the prediction results of whether network traffic is normal access or a network attack. In this way, the client can directly use the global feature statistics to complete local business decisions without updating the feature extraction module parameters or leaking local raw data, achieving collaborative unification of cross-organizational knowledge sharing and local task execution.

[0043] This invention also includes an asynchronous federated continuous learning method for large financial models, applied to servers, such as... Figure 2 As shown, it includes the following steps: S1': Initial model parameters of the global model are obtained by training based on public datasets or historical business data. The global feature extraction module is determined, and its parameters are frozen. The initial model parameters and the global feature extraction module are distributed to each client, enabling the client to represent the local business task data based on the global feature extraction module when processing local financial business tasks, and to construct corresponding feature prototypes and feature covariance statistics for different business categories. The client then asynchronously uploads incremental feature information, including feature prototypes and feature covariance statistics, to the server. At this time, the server does not require all clients to complete the upload at the same time, thereby avoiding synchronous blocking.

[0044] S2': Asynchronously receive incremental feature information from different clients.

[0045] S3': Asynchronous federated fusion of incremental feature information is performed to update global feature statistics, thereby mitigating the interference of new task knowledge on existing feature representations, reducing model oscillations caused by non-independent and identically distributed features, thus alleviating the problem of historical knowledge forgetting and maintaining the stability of global feature representations.

[0046] S4': Distribute the updated global feature statistics to each client to support the local decision-making modeling process for subsequent financial business tasks.

[0047] Combining the two embodiments described above yields a preferred embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps: S10: The server trains the initial model parameters of the global model based on public datasets or historical business data, determines the global feature extraction module, and the parameters of the global feature extraction module are frozen; the initial model parameters of the global model and the global feature extraction module are distributed to each client. S20: The client receives the initial model parameters of the global model and the global feature extraction module from the server. Based on the global feature extraction module, it performs feature representation on the local business task data and constructs corresponding feature prototypes and feature covariance statistics for different business categories to characterize the incremental knowledge acquired by the client in the current task stage. S30: The client uploads incremental feature information, including feature prototypes and feature covariance statistics, to the server asynchronously; S40: The server asynchronously receives incremental feature information from different clients, performs asynchronous federated fusion of feature prototypes and feature statistics in the incremental feature information, and updates the global feature statistics; S50: The server distributes the updated global feature statistics to each client for local decision-making modeling in subsequent client tasks; when a new financial business task arrives, steps S20 to S50 are repeated to achieve asynchronous federated continuous learning of the large financial model under a fixed parameter structure.

[0048] In one embodiment, the server trains a large financial model based on a public financial dataset or historical business data to obtain initial global model parameters and determines the global feature extraction module. ,include: Construct a set of learnable prototype vectors Each prototype vector A feature prototype corresponds to a category. Cosine similarity is used to calculate the similarity between the sample features and the prototype vector, yielding the classification probability. : in, Indicates that in a given input sample Under these conditions, the sample belongs to category The classification probability; This is a temperature coefficient used to control the smoothness of the probability distribution. A cross-entropy loss function is employed. Optimize the global feature extraction module: After training, the feature extractor Freezing is used for feature extraction in downstream tasks to ensure consistency in the feature space. The trained global model parameters and feature extractor are distributed to each financial institution's client, and each client retains a local copy of the model for generating feature representations of task data.

[0049] In one embodiment, the client performs feature representation on local business task data based on the global feature extraction module, specifically including: For the client Task Using the global feature extraction module Generate feature matrix : ; ; in, The number of samples; Indicates client In the The first task One input sample data; This represents the matrix transpose. To reduce the impact of client data distribution offset on feature representation and improve downstream task performance, the feature matrix is... Perform feature distribution calibration: ; in, The hyperparameters are used to calibrate and control the degree of characteristic distribution calibration; in this invention, the following parameters are taken: Post-calibration features It is closer to a Gaussian distribution and can be used for training downstream task classification or prediction models, improving the model's adaptability and accuracy in local tasks.

[0050] In one embodiment, the client constructs corresponding feature prototypes and feature covariance statistics for different business categories, including: (1) Calculate the mean of the feature vectors of each business category as the feature prototype; Specifically, the client The feature representations are divided according to business categories, and a corresponding feature prototype is built for each category. The feature prototype is defined as the mean vector of the features of samples in that category, and its calculation expression is: in, Indicates task Belongs to the category The sample set; Indicates the first task in the current client task. One input sample data; Indicates the first task in the current client task. The category label corresponding to each input sample.

[0051] (2) The feature prototype constructed by the current task is weighted and fused with the historical feature prototype according to the sample quantity ratio to obtain the fused and updated feature prototype.

[0052] Specifically, the feature prototype of the current category is weighted and fused with the feature mean vector of the previous time step t-1 to obtain the client-side result. The updated feature mean vector is calculated as follows: ; Where, the left side of the equation The updated feature mean vector; the left side of the equation The feature mean vector before the update; Indicates client Medium category From task 1 to... The cumulative sample size ratio weights for each task; Its corresponding feature prototype The cumulative required sample size ratio; after completing the above weighted fusion, the updated feature mean vector is determined as the client-side fused feature prototype, denoted as... This is used for subsequent uploading to the server to participate in global fusion.

[0053] (3) Calculate the covariance matrix based on the feature vectors of each business category sample, and use it as the feature covariance statistic; Specifically, in order to characterize the discreteness of the feature distribution of this category, the client... For each category, construct the feature covariance statistic: ; in, Indicates client In the Category of each task Characteristic covariance components.

[0054] (4) The feature covariance statistics constructed in the current task are weighted and fused with the historical feature covariance statistics according to the sample size ratio to obtain the fused and updated feature covariance statistics.

[0055] The feature covariance statistics constructed in the current task are then weighted and fused with the intra-class covariance from the previous time step t-1 to obtain the client. Updated intra-class covariance: ; Where, the left side of the equation The updated intra-class covariance; the right side of the equation The in-class covariance before the update; Its corresponding The required sample size ratio is used to balance the differences in data volume at different stages; Indicates client In the Category of each task From task 1 to... The cumulative sample size of each task is weighted proportionally; simultaneously, to characterize the distribution shift among clients, inter-class covariance is introduced: ; Where, the left side of the equation The updated inter-class covariance; the right side of the equation The inter-class covariance before the update; Indicates client In the Category of each task The inter-class covariance matrix; Indicates client In the The set of mean vectors of all class features in each task. Combining the within-class and between-class covariances yields the final fused covariance. : .

[0056] This embodiment leverages the temporal independence of class-level covariance to achieve continuous knowledge fusion on the client side. By continuously modeling intra-class and inter-class covariance, a second-order statistical representation of an approximate joint distribution is formed, effectively mitigating the interference caused by distribution shift.

[0057] In one embodiment, after the client constructs corresponding feature prototypes and feature covariance statistics for different business categories, it also includes: The quality and contribution of local data on the client side are quantitatively evaluated to generate a client quality score; Incremental feature information, including feature prototypes, feature covariance statistics, and client quality scores, is uploaded to the server asynchronously. The server then detects differences in client data distribution based on the currently stored global feature prototypes, constructs adaptive fusion weights by combining them with the client quality scores, and then performs asynchronous federated fusion of feature prototypes and feature covariance statistics based on the adaptive fusion weights to update the global feature statistics.

[0058] Specifically, the quality and contribution of local data on the client are quantitatively evaluated to generate a client quality score, including: (1) Calculate the consistency index between the current task data and the historical data distribution. The consistency index is calculated based on the Euclidean distance between the mean of current task features and the mean of historical fused features. It measures the consistency between the distribution of current task data and historical data. The calculation expression is: ; in The mean of the current task features; This represents the average of historical integration characteristics.

[0059] (2) Calculate the class coverage index of the client in the global label space in the current task. The category coverage metric is calculated based on the ratio of the size of the tag set involved in the current task to the size of the global tag set. It measures the client's ability to cover the global tag space, and the calculation expression is: ; in, This refers to the set of tags involved in the current task on the client side. This is a global tag set.

[0060] (3) Calculate data noise index based on trace of covariance matrix The data noise index is calculated based on the mean of the traces of the class covariance matrices. The class covariance traces are used to characterize data uncertainty, and the calculation expression is as follows:

[0061] (4) The consistency index, category coverage index and data noise index are weighted and summed to generate a client quality score. The calculation expression is: ; in, , , , These are the weighting coefficients for the consistency index, category coverage index, and data noise index, respectively. The client uploads the quality score simultaneously when uploading feature statistics. It is used for server-side execution of adaptive asynchronous federated fusion.

[0062] In one embodiment, the client uploads incremental feature information, including feature prototypes, feature covariance statistics, and client quality scores, to the server. At this time, the server performs the following actions: Asynchronously receive incremental feature information from different clients, including feature prototypes, feature covariance statistics, and client quality scores; The differences in client data distribution are detected based on the currently stored global feature prototypes. An adaptive fusion weight is constructed by combining the client quality score. Then, based on the adaptive fusion weight, the feature prototypes, feature covariance statistics, and global feature statistics are asynchronously federated and updated.

[0063] In one embodiment, after the server asynchronously receives incremental feature information from different clients, it further includes detecting differences in client data distribution based on the currently stored global feature prototype: (1) Calculate the distribution offset of the corresponding category based on the feature prototype uploaded by the client and the global feature prototype currently stored on the server. The calculation expression is as follows: in, Indicates the distribution offset of the category; Indicates the category uploaded by the client. Feature prototype; This represents the global feature prototype currently stored on the server. This represents the global covariance matrix currently stored on the server.

[0064] (2) Calculate the mean of the distribution offset of all business categories involved in the client, as the overall distribution offset of the client. The calculation expression is: in, Indicates the overall distribution offset of the client; Represents a set of categories Size.

[0065] (3) When the overall distribution shift Exceeding the preset threshold If a client is found to have a significant non-independent identically distributed offset, its fusion weight will be reduced in asynchronous federated fusion.

[0066] Furthermore, the server updates the global feature statistics according to a preset asynchronous fusion strategy for the relevant business categories. Based on the original sample proportion weights, a joint weighting mechanism for quality and distribution is introduced. Specifically, adaptive fusion weights are first calculated. : in, For sample proportion weights; Rate the client's quality; To adjust for the hyperparameters affected by distribution shift, the update forms for the global feature prototype and covariance are as follows: The asynchronous fusion mechanism described above can limit drastic shifts in the global feature space and allow each client to fully participate in the fusion update.

[0067] In one embodiment, after the client receives the merged and updated global feature statistics from the server, it further includes: Construct a covariance-aware Bayesian classifier based on global feature prototypes and the global covariance matrix; Extract the feature representation of newly arrived business task data, calculate the Mahalanobis distance between the feature representation and the global feature prototype of each category based on the covariance-aware Bayesian classifier, and classify the corresponding business task data into the category with the smallest Mahalanobis distance, thereby achieving accurate classification of business task data.

[0068] Specifically, after obtaining the global covariance and global prototype, a covariance-aware Bayesian classifier is introduced to transform statistical knowledge into a classification decision boundary. This is then applied to the corrected feature vectors. , Represents the space of real numbers. Represents the vector dimension, with assumptions about the category. The distribution is Gaussian: ; in, Represents a given category Under the condition, feature vector The probability density of occurrence; Indicate category The global feature prototype. Among them... , According to the optimal Bayesian decision rule: in, Represents a given feature vector Under the conditions, category The probability density of occurrence; Indicate category The prior probability; Representing the eigenvector The marginal probability. We can obtain: in, Represents the determinant of the covariance matrix; The Mahalanobis distance. Meanwhile, to ensure the invertibility of the covariance matrix and consistency with the metric space, we perform pre-metric processing: in, Indicates to Perform covariance reduction operation; Indicates the reduction factor; , These represent the mean diagonal variance and the mean off-diagonal variance, respectively. Represents the identity matrix; Indicates to Perform covariance normalization; Indicate category The normalized covariance matrix of the th Line 1 Column elements; Indicate category The covariance matrix of the first Line 1 Column elements; Represents the th class in the covariance matrix of class y. Standard deviation of dimension; Represents the th class in the covariance matrix of class y. Standard deviation of dimension.

[0069] Furthermore, a quadratic Bayesian decision boundary is constructed using fused covariance, adaptively reflecting the category-specific feature distribution. Pre-metric processing ensures matrix invertibility and maintains geometric consistency, thus achieving robust classification under asynchronous tasks and non-independent identically distributed conditions. (Setting...) Given the pre-metric covariance matrix, the Bayesian classification rule is: ; in, Indicates the prediction category.

[0070] Combining the above examples yields another preferred example of the asynchronous federated continuous learning method of the present invention, such as... Figure 4 As shown, the method includes the following steps: S100: The server trains the initial model parameters of the global model based on public datasets or historical business data, determines the global feature extraction module, and the parameters of the global feature extraction module are frozen; the initial model parameters of the global model and the global feature extraction module are distributed to each client. S200: The client receives the initial model parameters of the global model and the global feature extraction module from the server. Based on the global feature extraction module, it performs feature representation on the local business task data and constructs corresponding feature prototypes and feature covariance statistics for different business categories to characterize the incremental knowledge acquired by the client in the current task stage.

[0071] S300: Quantitatively evaluates the quality and contribution of local data on the client side, and generates a client quality score; S400: The client asynchronously uploads incremental feature information, including feature prototypes, feature covariance statistics, and client quality scores, to the server. S500: The server asynchronously receives incremental feature information from different clients, detects differences in client data distribution based on the currently stored global feature prototype, constructs adaptive fusion weights by combining client quality scores, and then performs asynchronous federated fusion of feature prototypes and feature covariance statistics based on the adaptive fusion weights to update global feature statistics. S600: The updated global feature statistics are distributed to each client for decision modeling of subsequent financial business tasks. The global feature statistics include the central position of each category (represented by the global feature prototype) and the distribution state (represented by the global covariance matrix). The client further builds a covariance-aware Bayesian classifier, which calculates Mahalanobis distance to classify and predict newly arrived business task data, such as determining whether a transaction is a normal transaction or a fraudulent transaction.

[0072] When a new financial business task arrives at any client, the client repeats steps S200 to S600 based on the latest global feature statistics that have been issued.

[0073] To address the common problems of existing federated learning methods for large-scale financial models in multi-institutional collaborative scenarios, such as strong synchronization dependencies, difficulty in adapting to inconsistent task arrival orders, and susceptibility to historical knowledge forgetting during continuous learning, this invention proposes an asynchronous federated continuous learning method for large-scale financial models. This invention introduces an asynchronous collaboration mechanism, a client-side data quality evaluation mechanism, and an adaptive fusion strategy that is aware of distribution differences into the federated learning framework. After obtaining the initial global model on the server side, it no longer performs centralized or traditional federated iterative training of model parameters. Instead, it continuously and asynchronously fuses incremental feature statistical information generated by clients from various financial institutions. This allows the global model to continuously absorb new knowledge from multi-institutional and multi-stage financial transactions while maintaining a stable parameter structure, achieving long-term evolution and stable improvement of model capabilities. In this method, each client first represents local financial business data based on a fixed global feature extraction module and constructs feature prototypes and covariance statistics that characterize local data distribution and decision-making information. Then, through a client data quality and contribution evaluation mechanism, the local data quality, category coverage, and noise level are quantitatively assessed. After receiving incremental feature information uploaded by the clients, the server performs adaptive asynchronous fusion based on client quality scores and data distribution difference detection results. This reduces the adverse impact of low-quality or distributionally offset clients on the global model, improving the model's stability and convergence efficiency in non-independent and identically distributed environments. This method is suitable for federated learning systems composed of server nodes and multiple financial institution clients. While ensuring that financial data remains within the domain, it enables different institutions to continuously collaborate in building and improving the feature representation and decision-making capabilities of large financial models despite inconsistent task sequences, varying computing power, and differences in data distribution. This enhances the model's adaptability and robustness in dynamic financial business scenarios.

[0074] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. An asynchronous federated continuous learning method for large models, characterized in that, When applied to the client, the following steps are included: The system receives the initial model parameters of the global model and the global feature extraction module from the server. The initial model parameters of the global model are obtained by the server based on public datasets or historical business data. The global feature extraction module is determined based on the initial model parameters of the large model, and the parameters of the global feature extraction module are in a frozen state. When processing local financial business tasks, the local business task data is represented by features based on the global feature extraction module, and corresponding feature prototypes and feature covariance statistics are constructed for different business categories; the feature prototype is the mean vector of the sample features of different business categories; the feature covariance statistics is the covariance matrix that reflects the distribution of sample features of different business categories. Incremental feature information, including feature prototypes and feature covariance statistics, is uploaded to the server asynchronously, enabling the server to perform asynchronous federated fusion of the incremental feature information and update the global feature statistics; the global feature statistics include global feature prototypes and global covariance matrix. The server receives updated global feature statistics, which are used for local decision modeling in subsequent tasks.

2. The asynchronous federated continuous learning method for large models according to claim 1, characterized in that, For different business categories, construct corresponding feature prototypes and feature covariance statistics, including: Calculate the mean of the feature vectors of samples from each business category as the feature prototype; The feature prototypes constructed for the current task are weighted and fused with the historical feature prototypes according to the sample size ratio to obtain the fused and updated feature prototypes. The covariance matrix is ​​calculated based on the feature vectors of samples from each business category, and used as the feature covariance statistic. The feature covariance statistics constructed for the current task are weighted and fused with the historical feature covariance statistics according to the sample size ratio to obtain the fused and updated feature covariance statistics.

3. The asynchronous federated continuous learning method for large models according to claim 1, characterized in that, After constructing corresponding feature prototypes and feature covariance statistics for different business categories, the following is also included: The quality and contribution of local data on the client side are quantitatively evaluated to generate a client quality score; Incremental feature information, including feature prototypes, feature covariance statistics, and client quality scores, is uploaded to the server asynchronously. The server then detects differences in client data distribution based on the currently stored global feature prototypes, constructs adaptive fusion weights by combining them with the client quality scores, and then performs asynchronous federated fusion of feature prototypes and feature covariance statistics based on the adaptive fusion weights to update the global feature statistics.

4. The asynchronous federated continuous learning method for large models according to claim 3, characterized in that, The quantitative evaluation of the quality and contribution of local data on the client side, generating a client quality score, includes: Calculate the consistency index between the current task data and the historical data distribution; Calculate the class coverage metric of the client in the global label space during the current task; Calculate data noise index based on covariance matrix trace; The consistency index, category coverage index, and data noise index are weighted and summed to generate a client quality score.

5. The asynchronous federated continuous learning method for large models according to claim 1, characterized in that, After the receiving server merges and updates the global feature statistics, it also includes: Construct a covariance-aware Bayesian classifier based on global feature prototypes and the global covariance matrix; Extract the feature representation of newly arrived business task data, calculate the Mahalanobis distance between the feature representation and the global feature prototype of each category based on the covariance-aware Bayesian classifier, and classify the corresponding business task data into the category with the smallest Mahalanobis distance.

6. An asynchronous federated continuous learning method for large models, characterized in that, When applied to a server, the following steps are included: The initial model parameters of the global model are obtained by training on public datasets or historical business data. The global feature extraction module is determined, and its parameters are frozen. The initial model parameters and the global feature extraction module are distributed to each client, enabling the client to represent the local business task data based on the global feature extraction module when processing local financial business tasks. Corresponding feature prototypes and feature covariance statistics are constructed for different business categories. The client also asynchronously uploads incremental feature information, including feature prototypes and feature covariance statistics, to the server. The feature prototype is the mean vector of sample features for different business categories, and the feature covariance statistics are the covariance matrix reflecting the distribution of sample features for different business categories. Asynchronously receive incremental feature information from different clients; Incremental feature information is asynchronously federated and fused to update global feature statistics; the global feature statistics include global feature prototypes and global covariance matrix. The updated global feature statistics are distributed to each client for local decision-making modeling in subsequent client tasks.

7. The asynchronous federated continuous learning method for large models according to claim 6, characterized in that, After asynchronously receiving incremental feature information from different clients, the method also includes detecting differences in client data distribution: Calculate the distribution offset of the corresponding category based on the feature prototype uploaded by the client and the global feature prototype currently stored on the server; Calculate the mean of the distribution offset of all business categories involved in the client, and use it as the overall distribution offset of the client; When the overall distribution offset exceeds a preset threshold, it is determined that the client has a significant non-independent identical distribution offset, and the client's fusion weight is reduced in asynchronous federated fusion.

8. The asynchronous federated continuous learning method for large models according to claim 6, characterized in that, The asynchronous federated fusion of incremental feature information to update global feature statistics also includes: Asynchronously receive incremental feature information from different clients; the incremental feature information includes feature prototypes, feature covariance statistics, and client quality scores, which are obtained by quantitatively evaluating the quality and contribution of the client's local data; The differences in client data distribution are detected based on the currently stored global feature prototypes. An adaptive fusion weight is constructed by combining the client quality score. Then, based on the adaptive fusion weight, the feature prototypes, feature covariance statistics, and global feature statistics are asynchronously federated and updated.

Images