Distributed big data anomaly detection system and method based on federated learning

Abstract

The invention relates to the technical field of big data anomaly detection systems, in particular to a federated learning-based distributed big data anomaly detection system and method, and the system comprises a dynamic feature extractor, a federated anomaly detector, a federated anomaly interpreter, a federated anomaly response coordinator, and a federated learning optimizer. The dynamic feature extractor is used for continuously monitoring a real-time data stream and extracting features to form feature vectors; the federal anomaly detector is used for judging whether the monitoring data flow is abnormal or not; the federal anomaly interpreter is used for quantifying feature importance and analyzing the cause of the anomaly; the federal exception response coordinator is used for coordinating response strategies of corresponding participants according to exception reasons; the federal learning optimizer adjusts system parameters according to an abnormal response strategy result, and realizes efficient, safe and extensible abnormal detection through innovative dynamic feature extraction, multi-model fusion and an adaptive response strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of big data anomaly detection systems, and in particular to a distributed big data anomaly detection system and a method thereof based on federated learning. Background Art

[0002] In today's digital age, with the rapid development of big data technology, anomaly detection has become an indispensable tool in all walks of life. Especially in high-risk fields such as finance, medical care, and network security, effective anomaly detection systems can prevent fraud, identify potential risks, and even save lives. However, with the continuous expansion of data scale and the increasing diversity of data sources, traditional anomaly detection methods have gradually revealed their limitations.

[0003] At present, the anomaly detection methods commonly used in the industry are mainly divided into two categories: centralized methods and simple distributed methods. The centralized method requires that all data be gathered to a central server for analysis. Although this method can fully utilize data information, it faces severe challenges in practical applications. First, the centralized storage and processing of data brings huge risks of privacy leakage, which becomes particularly difficult under the current increasingly stringent data protection regulations. Secondly, with the exponential growth of data volume, the computing burden of the central server is becoming increasingly heavy, resulting in a decrease in system response speed, which is difficult to meet the needs of real-time monitoring. In addition, the security risks and bandwidth pressure during data transmission cannot be ignored.

[0004] On the other hand, although the simple distributed approach alleviates the privacy issue to a certain extent, it also brings new dilemmas. In this approach, each participant independently runs its own anomaly detection model, and then integrates the results through simple voting or averaging mechanisms. Although this method protects the original data, it is difficult to fully utilize global information, resulting in a significant decrease in detection accuracy. At the same time, the data quality and model performance of different participants vary greatly, and simple result integration is difficult to balance these differences, further affecting the detection effect.

[0005] More importantly, existing methods generally lack the ability to adapt to dynamic environments. In practical applications, abnormal patterns are often constantly evolving, and new fraud methods emerge in an endless stream. Static detection models are unable to capture these changes in a timely manner, causing the effectiveness of the system to decline rapidly over time. In addition, existing methods also face challenges in dealing with highly heterogeneous data sources, and it is difficult to effectively integrate data from different fields and in different formats.

[0006] In view of the above problems, new anomaly detection methods are urgently needed, which can fully utilize the global information in distributed environments while protecting data privacy, and have the ability to dynamically adapt and process heterogeneous data. Summary of the invention

[0007] The purpose of the present invention is to provide a distributed big data anomaly detection system and method based on federated learning to solve key problems in the prior art, such as insufficient data privacy protection, insufficient global information utilization, poor dynamic adaptability, and weak heterogeneous data processing capabilities.

[0008] The present invention provides a distributed big data anomaly detection system based on federated learning, comprising: a dynamic feature extractor, a federated anomaly detector, a federated anomaly interpreter, a federated anomaly response coordinator, and a federated learning optimizer; the dynamic feature extractor is used to continuously monitor real-time data streams and extract features to form feature vectors; the federated anomaly detector is used to determine whether the monitored data stream is abnormal; the federated anomaly interpreter is used to quantify the importance of features and analyze the causes of anomalies; the federated anomaly response coordinator is used to coordinate the response strategies of corresponding participants according to the causes of anomalies; the federated learning optimizer adjusts system parameters according to the results of the anomaly response strategy.

[0009] The anomaly detection method of the distributed big data anomaly detection system based on federated learning includes the following specific method steps:

[0010] Step 1: Dynamically extract data features of participants; extract data features of participants in real time through two levels of heterogeneous network embedding and multi-feature extraction;

[0011] Step 2: Anomaly scoring and anomaly knowledge distillation: Split the federated learning model based on time series, space and overall weights, and evaluate the anomaly threshold of the intermediate model; Strengthen it according to the potential knowledge fusion method and output the final result report;

[0012] Step 3: Generate feature explanatory reports; use explanatory techniques and explanatory reports to promote real-time updates of anomaly detection results;

[0013] Step 4: Set the response strategy; design an adaptive response strategy based on the dynamic feature interpretation report;

[0014] Step 5: Strengthen the anomaly detection system; adjust the system anomaly detection related parameters according to the adaptive response strategy.

[0015] Advantages and beneficial effects of the present invention:

[0016] The distributed big data anomaly detection system based on federated learning proposed in the present invention realizes efficient, secure and scalable anomaly detection through innovative dynamic feature extraction, multi-model fusion and adaptive response strategy. This method can not only make full use of global information under the premise of protecting the data privacy of all parties, but also continuously optimize the detection model according to real-time feedback to adapt to the ever-changing anomaly patterns. At the same time, the method of the present invention also has the ability to process highly heterogeneous data and can effectively integrate diversified data from different sources. In general, the present invention provides a new solution for large-scale anomaly detection in complex environments, which has significant advantages in accuracy, privacy protection, adaptability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a logic block diagram of the system of the present invention.

[0018] Figure 2 It is a logic block diagram of the dynamic feature extractor of the present invention.

[0019] Figure 3 It is a logical block diagram of the federated anomaly detector of the present invention.

[0020] Figure 4 It is a logical block diagram of the federal exception interpreter of the present invention.

[0021] Figure 5 It is a logic block diagram of the federal exception response coordinator of the present invention.

[0022] Figure 6 This is a logical block diagram of the federated learning optimizer of the present invention. DETAILED DESCRIPTION

[0023] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0024] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0025] See also Figure 1-6The present invention discloses a distributed big data anomaly detection system based on federated learning, including: a dynamic feature extractor 1, a federated anomaly detector 2, a federated anomaly interpreter 3, a federated anomaly response coordinator 4, and a federated learning optimizer 5; the dynamic feature extractor 1 is used to continuously monitor the real-time data stream and extract features to form a feature vector; the federated anomaly detector 2 is used to determine whether the monitored data stream is abnormal; the federated anomaly interpreter 3 is used to quantify the importance of features and analyze the causes of anomalies; the federated anomaly response coordinator 4 is used to coordinate the response strategies of corresponding participants according to the causes of anomalies; the federated learning optimizer 5 adjusts system parameters according to the results of the anomaly response strategy.

[0026] The present invention cleverly combines federated learning with anomaly detection and proposes an innovative distributed anomaly detection system.

[0027] First, the dynamic feature extractor 1 acts as the eyes of the system, continuously monitoring the real-time data stream and extracting key features from it. In practical applications, such as in the scenario of a large e-commerce platform, the dynamic feature extractor 1 may focus on multi-dimensional information such as users' browsing behavior, purchasing patterns, and payment methods. This information is converted into feature vectors, laying the foundation for subsequent anomaly detection. A significant advantage of the dynamic feature extractor 1 is that it can adapt to real-time changes in data and capture emerging patterns and features in a timely manner.

[0028] Secondly, the Federated Anomaly Detector 2 plays the role of the brain of the system. It uses the feature vectors collected from various participants to determine whether there are anomalies in the data stream. For example, in an e-commerce platform, it may find that a user's purchasing behavior suddenly changes dramatically, or that there are transactions that are seriously inconsistent with historical patterns. The advantage of the Federated Anomaly Detector 2 is that it can use multi-party data for more comprehensive and accurate anomaly detection while protecting data privacy.

[0029] Third, the Federated Anomaly Explainer 3 acts as an analyst. It not only points out anomalies, but also deeply analyzes the causes of anomalies. For example, it may find that the main factor causing the anomaly is that the user tried to use different payment methods multiple times in a short period of time. This explainability is extremely valuable to system administrators and security teams, helping them to quickly locate problems and take appropriate measures. An important advantage of the Federated Anomaly Explainer 3 is that it provides explainability of anomaly detection, which is particularly important in many highly regulated industries (such as finance and healthcare).

[0030] Then, the Federal Anomaly Response Coordinator 4 is the commander of the system. Based on the analysis results of the Anomaly Interpreter, it coordinates the various participants to adopt appropriate response strategies. In the example of an e-commerce platform, this may mean temporarily freezing suspicious accounts, adding additional authentication steps, or sending alerts to the security team. The advantage of the Federal Anomaly Response Coordinator 4 is that it can coordinate the actions of multiple parties in a distributed environment to ensure the consistency and effectiveness of the response.

[0031] Finally, the Federated Learning Optimizer 5, as the system learns, continuously adjusts system parameters based on the effect of the anomaly response. This ensures that the system can keep pace with the times and adapt to changing anomaly patterns. A key advantage of the Federated Learning Optimizer 5 is that it can optimize the model using feedback from multiple parties without directly sharing the original data, which not only protects data privacy but also improves the generalization ability of the model.

[0032] The dynamic feature extractor 1 includes: a multi-feature extraction module 11, which is used to obtain real-time data of participants and extract multi-dimensional feature vectors through time series and space modeling methods; a detector selection module 12, which dynamically selects anomaly detectors based on dynamically changing weights to continuously optimize detection efficiency under limited resources; the multi-feature extraction module 11 includes a multi-scale spatiotemporal convolution model and a heterogeneous network embedding model; the multi-scale spatiotemporal convolution model collects and aggregates local spatiotemporal features in a non-overlapping manner at each scale to extract multi-granularity spatiotemporal features; by fusing the features of each sampling scale, a spatiotemporal feature vector is formed; the input of the heterogeneous network embedding is the spatiotemporal features extracted by the previous module, and by constructing a local spatiotemporal feature graph, using edge convolution to aggregate adjacent node features, and then performing feature mapping from structure to node.

[0033] Now, let’s take a closer look at the internal structure of the dynamic feature extractor 1. The dynamic feature extractor 1 consists of two core modules: a multi-feature extraction module 11 and a detector selection module 12. These two modules work together to ensure that the system can efficiently extract key features from massive data and select the most suitable anomaly detector.

[0034] First, the multi-feature extraction module 11 is responsible for extracting multi-dimensional feature vectors from the real-time data of the participants. This process involves complex temporal and spatial modeling methods. For example, in the field of network security, the multi-feature extraction module 11 may simultaneously analyze the temporal pattern of network traffic (such as sudden increases in traffic during certain time periods) and spatial features (such as the geographical distribution of source IP addresses).

[0035] One of the innovations of the present invention is the use of a multi-scale spatiotemporal convolutional model and a heterogeneous network embedding model. The multi-scale spatiotemporal convolutional model can capture features of different time spans and spatial ranges. For example, it can simultaneously focus on the data patterns of the last 5 minutes, the last hour, and the last 24 hours, thereby fully grasping the spatiotemporal characteristics of the data. The advantage of this multi-scale approach is that it can simultaneously capture short-term fluctuations and long-term trends, improving the accuracy and robustness of anomaly detection.

[0036] Specifically, the multi-scale spatiotemporal convolution model collects and aggregates local spatiotemporal features in a non-overlapping manner at each scale to extract multi-granular spatiotemporal features. This process can be expressed by the following formula:

[0037]

[0038] Among them, l represents the number of convolutional layers, N represents the length of the input feature vector, represents the kth time series; The multi-granularity spatiotemporal features are extracted. By fusing the features of each sampling scale, a comprehensive spatiotemporal feature vector is finally formed.

[0039] The heterogeneous network embedding model further improves the system's expressiveness. It takes the spatiotemporal features extracted in the previous step as input, constructs a local spatiotemporal feature map, then uses edge convolution to aggregate adjacent node features, and finally performs feature mapping from structure to node. The advantage of this method is that it can effectively capture the complex relationships and dependencies between data, and is particularly suitable for processing complex heterogeneous network data such as social networks and financial transaction networks.

[0040] Secondly, the detector selection module 12 dynamically selects anomaly detectors based on dynamically changing weights. This module is designed to solve a key problem in practical applications: how to select the most appropriate anomaly detector to optimize detection efficiency when computing resources are limited. The detector selection module 12 achieves the best balance between accuracy and efficiency by continuously evaluating the performance of each detector and dynamically adjusting the weights according to the current data characteristics.

[0041] For example, in a financial transaction monitoring system, the detector selection module 12 may dynamically select different anomaly detectors according to different time periods (such as transaction peaks vs. troughs) and different types of transactions (such as large transactions vs. small transactions). During peak transaction periods, it may select a detector with faster calculation speed but lower accuracy to ensure real-time response; while during trough transaction periods, it may select a more complex and more accurate detector to improve the accuracy of anomaly detection.

[0042] A significant advantage of this dynamic selection mechanism is that it can adapt to changes in data distribution and fluctuations in system resources, always maintaining the best detection effect. At the same time, it also provides good scalability for the system, allowing new anomaly detection algorithms to be easily integrated without significantly modifying the existing system architecture.

[0043] The multi-feature extraction module 11 constructs a temporal feature graph through the spatiotemporal feature vector, aggregates the features of adjacent nodes using the edge convolution model, calculates the mapping from structure to node through the self-attention model, and combines the edge convolution and self-attention layers to realize the splicing and supplementation of spatiotemporal features and heterogeneous network features to complete feature extraction.

[0044] Now, let's take a closer look at the working principle of the multi-feature extraction module 11. The core task of this module is to extract high-quality feature representations from raw data through a series of complex mathematical models and algorithms. This process involves multiple steps, each of which contributes unique information to the final feature representation.

[0045] First, the multi-feature extraction module 11 constructs a time feature graph using the previously generated spatiotemporal feature vectors. The purpose of this step is to convert the time series data into a graph structure that is easier to process and analyze. For example, in a network traffic analysis system, a time feature graph may represent the traffic change relationship between different time points, where nodes represent different time points and edges represent the traffic change between adjacent time points.

[0046] Next, the system uses the edge convolution model to aggregate the features of adjacent nodes. Edge convolution is an important operation in graph neural networks, which can effectively utilize the relationship information between nodes. In the scenario, edge convolution can help the system capture local patterns and trends in time series. For example, it may find that certain specific time patterns (such as the daily morning peak) have a strong correlation with the occurrence of abnormal events. The calculation process of edge convolution can be expressed by the following formula:

[0047]

[0048] in, and Represent the features of nodes i and j respectively, N(i) represents the set of adjacent nodes of node i, and MLP is a multi-layer perceptron. The advantage of this operation is that it can adaptively learn the relationship between nodes without pre-defining the structure of the graph.

[0049] Then, the system introduces a self-attention model to calculate the mapping from structure to node. The introduction of the self-attention mechanism enables the system to automatically learn the importance weights between different time points or different features, so as to better capture long-term dependencies. This is particularly useful when processing long sequence data, such as when analyzing long-term user behavior patterns. The calculation of the self-attention mechanism can be expressed by the following formula:

[0050]

[0051] Among them, Q, K, and V represent query, key, and value matrices respectively. is the dimension of the key vector.

[0052] Finally, the system cleverly combines edge convolution and self-attention layers to achieve the splicing and complementation of spatiotemporal features and heterogeneous network features. The advantage of this combination is that it not only retains local structural information (through edge convolution) but also captures global dependencies (through self-attention), thus obtaining a more comprehensive and rich feature representation.

[0053] This multi-level, multi-angle feature extraction method enables the system to extract highly informative feature representations from raw data, laying a solid foundation for subsequent anomaly detection tasks. Especially when dealing with complex, large-scale distributed data, this method can effectively capture subtle patterns and complex relationships in the data, greatly improving the accuracy and interpretability of anomaly detection.

[0054] The detector selection module 12 estimates the dynamic weight based on the characteristic distribution information of each participant's local data stream detector performance statistics, and dynamically selects the detector according to the degree of influence on the abnormal detection result.

[0055] The detector selection module 12 is another innovative feature of the present invention. It optimizes system performance by dynamically selecting the most suitable anomaly detector. The core task of the detector selection module 12 is to dynamically select the most suitable anomaly detector according to the real-time data characteristics and system resource status during system operation. This dynamic selection mechanism greatly improves the adaptability and efficiency of the system. Let's take a deeper look at the working principle of this module.

[0056] First, the detector selection module 12 continuously monitors the local data stream of each participant and analyzes its feature distribution information. In this process, the system considers features of multiple dimensions, such as the temporal distribution, spatial distribution, and numerical range of the data. For example, in a financial transaction monitoring system, it may focus on the intraday changes in transaction frequency, the distribution of transaction amounts, and the geographical distribution of users. Through this comprehensive feature analysis, the system can have a deep understanding of the characteristics of the current data.

[0057] Secondly, based on these feature distribution information, the detector selection module 12 will perform statistical analysis on the performance of each anomaly detector. The "performance" here includes not only the detection accuracy, but also multiple aspects such as computational efficiency and resource consumption. For example, some detectors may perform well when processing highly skewed data distributions, while others may have an advantage when processing multimodal data. Through long-term performance statistics, the system can establish a comprehensive performance profile of the detector.

[0058] Next, the detector selection module 12 estimates the dynamic weight of each detector based on these performance statistics. This weight estimation process takes into account multiple factors such as the detector's historical performance, current data characteristics, and system resource status. This weight estimation process can be expressed by the following formula:

[0059] W i =α*AccuracyScore i +β*EfficiencyScore i +γ*ResourceScore i

[0060] Among them, W i Represents the weight of the i-th detector, AccuracyScore i 、EfficiencyScore i and ResourceScore i They represent the scores of the detector in terms of accuracy, efficiency and resource consumption, respectively, and α, β, and γ are the corresponding weight coefficients. These coefficients can be dynamically adjusted according to the specific needs of the system.

[0061] For example, when the system is heavily loaded, the values ​​of β and γ may be increased to give more consideration to efficiency and resource consumption; whereas when the data quality is poor or the abnormal pattern is complex, the value of α may be increased to place more emphasis on detection accuracy.

[0062] Finally, the detector selection module 12 will sort all available detectors according to the estimated weights and select the detector that best suits the current situation. This selection process is not a one-time process, but an ongoing process. The system will continuously adjust and optimize the selection strategy based on the feedback of the detection results.

[0063] This dynamic selection mechanism brings many advantages. First, it greatly improves the adaptability of the system. No matter how the data distribution changes, the system can select the most appropriate detector and maintain efficient anomaly detection capabilities. Second, it optimizes resource utilization. When resources are limited, the system can weigh accuracy and efficiency and select the optimal combination of detectors. Third, this mechanism facilitates the introduction of new detection algorithms into the system. New detectors can be easily integrated into the system and their usage frequency can be dynamically adjusted based on actual operating results.

[0064] For example, in a network security monitoring system, the detector selection module 12 may select different detectors at different time periods. During the nighttime period when network traffic is low, it may select some detectors with higher computational complexity but higher accuracy for in-depth analysis; and during the daytime peak period, it may select some lightweight but fast-response detectors to ensure real-time responsiveness.

[0065] The dynamic weight is dynamically adjusted based on the influence of the anomaly detection results, and the calculation method is:

[0066]

[0067] Where j represents the serial number of the federated learning participant, d represents the number of local convolutional neural network layers in the federated learning, is the number of convolution kernel weights corresponding to the local convolutional neural network layer d of the jth participant, is the kth convolution kernel weight, is the local detector result of the jth participant, is the federated learning result, is the difference between the local detector result of the jth participant and the federated learning result; the dynamic weight of the number of local convolutional neural network layers d in the federated learning can be expressed as:

[0068]

[0069] Now, we will discuss in depth the calculation method of dynamic weight and its important role in the system. The introduction of dynamic weight is a key innovation of the present invention, which enables the system to dynamically adjust the importance of each participant in the federated learning process according to their contribution.

[0070] First, let's analyze the calculation formula of this dynamic weight. Represents the weight of the jth participant in the dth layer of the convolutional neural network. This weight is determined by the product of two factors: the sum of the absolute values ​​of the convolution kernel weights, and the gap between the local detection result of the participant and the overall result of the federated learning.

[0071] Specifically, It represents the sum of the absolute values ​​of all convolution kernel weights of the jth participant in the dth layer of the network. This term reflects the complexity or expressiveness of the participant in a specific network layer. The larger the absolute value of the weight, the more important the layer of the network is in the feature extraction process.

[0072] It indicates the gap between the local test results of the participant and the overall results of federated learning. This gap reflects the contribution of the participant to the overall model. If the local results of a participant are significantly different from the overall results, it may mean that it has some unique information that is valuable to the overall model.

[0073] The product of these two items cleverly combines the model complexity of the participants and their contribution to the whole, forming a comprehensive weight indicator.

[0074] Next, by summing up the weights of all participants ( ), the overall weight of the entire federated learning system at a specific network layer is obtained. This overall weight reflects the importance of this layer in the entire federated learning process.

[0075] This dynamic weight calculation method brings several significant advantages:

[0076] 1. Adaptability: The system can dynamically adjust the importance of each participant based on its real-time performance, so that the federated learning process can better adapt to changes in data distribution and differences in the capabilities of the participants.

[0077] 2. Fairness: By considering the contribution of each participant, this approach ensures that truly valuable information can be given more attention in the federated learning process, rather than relying solely on the amount of data of the participants.

[0078] 3. Efficiency improvement: By dynamically adjusting weights, the system can pay more attention to participants who are more helpful in improving model performance, thereby accelerating the convergence process of the model.

[0079] 4. Enhanced anomaly detection capability: Since the weight calculation takes into account the difference between local results and overall results, this method can better capture local anomalies and improve the overall anomaly detection capability.

[0080] For example, in a cross-industry financial risk monitoring system, different participants (such as banks, insurance companies, securities companies, etc.) may face different types of financial risks. Through this dynamic weighting mechanism, the system can dynamically adjust the importance of each industry in the overall model according to its risk characteristics and detection capabilities at different times. For example, during economic turmoil, more reliance may be placed on the data and models of participants who are more sensitive to market fluctuations.

[0081] In general, this dynamic weight calculation method greatly improves the flexibility and effectiveness of the federated learning system in anomaly detection tasks, enabling the system to maximize the use of valuable information in distributed data while protecting data privacy.

[0082] The selection index of the dynamic selection detector is the maximum value of the vector and the maximum chi-square correlation value of the convolution kernel, and the calculation formula is:

[0083]

[0084] in, Represents the vector corresponding to the i-th convolution kernel, K is the number of convolution kernels, is the eigenvector, F is the number of vectors, represents the chi-square correlation calculation function, Indicates the similarity between the i-th convolution kernel and the overall maximum vector, select as a selection indicator.

[0085] This is another key innovation of the present invention. This selection method cleverly utilizes the chi-square correlation, allowing the system to efficiently select the detector that best suits the current data characteristics. The principle and advantages of this method are analyzed in depth.

[0086] First, the core idea of ​​this method is to select the most suitable detector by comparing the correlation between the feature vector and the convolution kernel. ) is a widely used statistical method to measure the degree of correlation between two variables. Here, it is used to measure the similarity between the feature vector and the convolution kernel.

[0087] In the formula represents the feature vector, Represents the vector corresponding to the i-th convolution kernel. The system calculates the chi-square correlation between each feature vector and each convolution kernel, and then selects the largest correlation value as the score of the convolution kernel. Finally, the convolution kernel with the highest score is selected as the final selection indicator among all convolution kernels.

[0088] This approach has several significant advantages:

[0089] 1. Efficiency: The calculation of chi-square correlation is relatively simple and fast, enabling the system to make decisions quickly in large-scale data and complex models

[0090] 2. Adaptability: By comparing the correlation between feature vectors and convolution kernels, the system can automatically adapt to different types of data distributions and abnormal patterns.

[0091] 3. Robustness: Selecting the maximum correlation value as the indicator can effectively capture the main features in the data while reducing the impact of noise.

[0092] 4. Interpretability: Chi-square correlation is an intuitive statistical indicator, which makes the selection process more interpretable, which is very important in practical applications.

[0093] Let’s use a specific example to illustrate the application of this method. Suppose in a network security monitoring system, there are multiple detectors, each of which corresponds to a set of convolution kernels. The feature vector received by the system may contain information such as network traffic, packet size, connection duration, etc.

[0094] When new data arrives, the system calculates the chi-squared correlation between these feature vectors and the convolution kernels of each detector. For example, if a detector’s convolution kernel has a high correlation with feature vectors representing abnormally large packets, then this detector may be selected for detecting possible DDoS attacks.

[0095] At the same time, this method also allows the system to adjust dynamically. If the network environment changes, causing the distribution of feature vectors to change, the system will automatically select a new detector that is more suitable for the current environment. This adaptive capability is particularly important in complex and changing network environments.

[0096] In addition, the selection method is computationally efficient, which means the system can respond quickly to changes in data flows. In high-speed network environments, this fast response capability can be key to detecting and defending against network attacks.

[0097] In general, this chi-square correlation-based detector selection method provides the system with a flexible, efficient, and interpretable way to dynamically select the most suitable anomaly detector. It enables the system to quickly adapt to the changing data environment while maintaining high detection accuracy, which is a very valuable feature in practical large-scale distributed systems.

[0098] The present invention also discloses an anomaly detection method of a distributed big data anomaly detection system based on federated learning, including the following specific method steps:

[0099] Step 1: Dynamically extract data features of participants; extract data features of participants in real time through two levels of heterogeneous network embedding and multi-feature extraction;

[0100] Step 2: Anomaly scoring and anomaly knowledge distillation: Split the federated learning model based on time series, space and overall weights, and evaluate the anomaly threshold of the intermediate model; Strengthen it according to the potential knowledge fusion method and output the final result report;

[0101] Step 3: Generate feature explanatory reports; use explanatory techniques and explanatory reports to promote real-time updates of anomaly detection results;

[0102] Step 4: Set the response strategy; design an adaptive response strategy based on the dynamic feature interpretation report;

[0103] Step 5: Strengthen the anomaly detection system; adjust the system anomaly detection related parameters according to the adaptive response strategy.

[0104] The method of heterogeneous network embedding and spatiotemporal feature extraction in step 1 is:

[0105] Step 1) Extract continuous value features and category features from the data stream of the participants, generate a time feature vector based on the time series features, collect sampled local features at each scale, and aggregate them at multiple sampling scales in this form; aggregate features from multiple discrete scales to extract multi-granularity spatiotemporal features; the aggregation formula of the lth layer is as follows:

[0106]

[0107] Where N represents the length of the input feature vector, represents the kth time series; For the extracted multi-granularity spatiotemporal features;

[0108] Step 2) Use multi-head attention and average attention layers to calculate the similarity of network node features; after aggregating adjacent node features through edge convolution, map from structure to node features to obtain the final node embedding; the similarity calculation formula of the average attention layer is as follows:

[0109]

[0110] Among them, W represents the input similarity matrix, Q, K, and V represent query, key, and value matrices respectively, d_k represents the number of input matrix nodes, softmax is the attention head, and S represents the output of the input through the multi-head attention mechanism; the calculation formula of the multi-head attention mechanism is:

[0111]

[0112]

[0113] in, , and represents the attention weight matrix of different heads, and h represents the number of attention heads;

[0114] Step 2-1, node neighbor representation based on attention mechanism:

[0115]

[0116] in, represents the node embedding vector of node i, represents the attention weight between nodes i and j, N(i) represents the set of adjacent nodes of node i, W is the weight matrix, For the activation function step 2-2, aggregate the adjacent node features through the edge convolution method to calculate the final node embedding:

[0117]

[0118] in, and Represent the features of nodes i and j respectively, N(i) represents the set of adjacent nodes of node i, and MLP is a multi-layer perceptron;

[0119] Step 2-3: Use multi-scale spatiotemporal features as temporary embedding to construct network features; the final node embedding is and Cascade system:

[0120]

[0121] Step 2-4: Embed the node As the feature representation of the local data flow, node features are exchanged between participants and inter-node features are output.

[0122] The method of heterogeneous network embedding and spatiotemporal feature extraction,First, the system extracts continuous value features and categorical features from the data streams of the participants and generates a temporal feature vector. This multi-scale feature extraction method can capture the characteristics of data at different temporal and spatial scales, and is very suitable for processing complex big data environments. For example, in an intelligent transportation system, this method can simultaneously capture short-term traffic flow fluctuations and long-term traffic pattern changes.

[0123] Secondly, the system uses a multi-head attention mechanism to calculate the similarity of network node features. The advantage of this method is that it can adaptively learn the importance weights between different features, and is particularly suitable for processing data with complex internal relationships. In the financial risk monitoring system, this method can effectively capture the complex correlations between different financial indicators.

[0124] Then, by aggregating neighboring node features through edge convolution, the system is able to effectively utilize graph structure information. This is particularly useful when processing data with network structure, such as social network analysis or supply chain risk management.

[0125] Finally, the system combines multi-scale spatiotemporal features with network features to obtain the final node embedding. This comprehensive feature representation can comprehensively describe all aspects of the data and provide strong support for subsequent anomaly detection.

[0126] In general, this heterogeneous network embedding and spatiotemporal feature extraction method is able to process complex, multi-source big data and extract rich and meaningful features, providing a solid foundation for anomaly detection systems.

[0127] The calculation strategy of the anomaly threshold evaluation in step 2 is: obtain the anomaly detection results from the participants, set the threshold θ to sort the participants; use the intermediate model to select the participants, and input the kth participant data into the intermediate result model, and determine the anomaly score by judging the anomaly type of the intermediate result model; the anomaly evaluation score calculation formula is as follows:

[0128]

[0129] in, is the threshold corresponding to the classifier i of the intermediate result model, k represents the serial number of all participants, n is the number of participants, represents the abnormal score corresponding to the kth participant, represents the feature extractor output corresponding to the kth participant, is the i-th intermediate result model pair The output of , I() is the indicator function;

[0130] The potential knowledge fusion method is used to integrate the intermediate result models of k participants, and F is used to fuse and strengthen the normal classifier and the abnormal classifier. The calculation formulas are:

[0131]

[0132]

[0133] Among them, N and P represent the activation function normal classifier and the activation function abnormal classifier respectively, and F is the fusion function. The calculation method of fusing the intermediate result model and strengthening the anomaly detection task is expressed as follows:

[0134]

[0135] in, and The final activation functions for normal and abnormal detection are and is the weight, b is the bias term, E is the intermediate result model, and R is the final detection result;

[0136] Step 2 is enhanced according to the potential knowledge fusion method, reasonably allocates the participants, and outputs the federated anomaly detection result report.

[0137] The calculation strategy and knowledge fusion method of anomaly threshold evaluation,First, the system obtains anomaly detection results from each participant and sets a threshold,θ,to sort the participants.,This method can effectively identify participants with abnormal performance,,providing important clues for subsequent anomaly detection.

[0138] The system then uses the intermediate model to evaluate the data of the participants and calculate the anomaly score. The advantage of this method is that it combines the judgments of multiple intermediate models and improves the robustness of anomaly detection. For example, in a multi-bank anti-money laundering system, this method can comprehensively consider the anomaly detection models of different banks to obtain a more comprehensive and accurate anomaly score.

[0139] Next, the system uses a latent knowledge fusion method to integrate the intermediate result models of multiple participants. The innovation of this method is that it can fuse the knowledge of multiple parties without directly sharing the original data. This can protect data privacy while making full use of the information in distributed data, which is very suitable for anomaly detection tasks in the financial or medical fields that require high confidentiality.

[0140] Finally, the system obtains the final detection result by fusing the results of normal classifier and abnormal classifier. This method can balance the influence of normal samples and abnormal samples and improve the accuracy and reliability of detection.

[0141] In general, this anomaly threshold assessment and knowledge fusion method fully utilizes the advantages of federated learning to achieve efficient and accurate anomaly detection while protecting data privacy. It is particularly suitable for scenarios that require multi-party collaboration but protect the data privacy of all parties, such as cross-industry financial risk monitoring or multi-institutional medical anomaly detection.

[0142] The steps for generating the feature interpretation report in step 3 are as follows:

[0143] Step 1: Generate a feature contribution graph using the interpretation method: Calculate the feature contribution and generate a ranking graph of the feature’s influence on the overall performance for participant selection. The formula is as follows:

[0144]

[0145] in, is the importance of feature j, is the model output for the original input x, is the model output after removing feature j;

[0146] Step 2-1: Obtain the feature contribution matrix through the maximum difference method:

[0147]

[0148] in, and Represent the importance mean and standard deviation of feature i respectively, The standard Euclidean distance between the feature vector after removing variable j for training corresponding feature i and all feature vectors to be evaluated Step 2-2, further calculate through abnormal contribution to obtain the feature contribution vector:

[0149]

[0150] in, is the importance of feature i, is the mean importance value of all features in the test set, is the standard deviation;

[0151] Step three generates a visualization report to facilitate the user's understanding of the decision-making process of the anomaly detection model; the visualization report includes: (1) an explanation of the steps for calculating feature contribution; and (2) an explanation of the description of the anomaly feature.

[0152] The generation process of feature explanatory reports is an important innovation of this invention. In a complex big data environment, it is not enough to just get the anomaly detection results, but also to understand why these anomalies occur. This interpretability is of great significance for practical applications, especially in high-risk fields such as finance and medicine.

[0153] First, the system uses interpretation techniques to generate a feature contribution graph. The core of this step is to calculate the contribution of each feature to the model output. The method used in this paper is to quantify the importance of features by comparing the difference between the original model output and the model output after removing a certain feature. This method is intuitive and effective, and can clearly show which features have the greatest impact on the anomaly detection results.

[0154] For example, in a credit card fraud detection system, this step might reveal that "transaction amount" and "transaction location" are the most important factors in determining whether a transaction is fraudulent. This information not only helps understand the model's decision-making process, but may also provide valuable insights for fraud prevention.

[0155] Secondly, the system obtains the feature contribution matrix through the maximum difference method. This step further refines the relative importance of the features. By calculating the mean and standard deviation of the feature importance, the system can more comprehensively evaluate the contribution of each feature. The advantage of this method is that it considers the distribution of feature importance, not just the average, thus providing richer information.

[0156] The system then calculates the anomaly contribution and obtains the feature contribution vector. This step compares the importance of each feature with the overall average, highlighting those features that are particularly important for anomaly detection. This standardized approach makes the importance of different features more comparable.

[0157] Finally, the system generates a visual report. This report not only contains the calculation process of feature contribution, but also provides a detailed explanation of abnormal features. The advantage of this visual report is that it can convert complex mathematical models into a form that humans can understand. For example, the report may show the importance ranking of each feature in the form of a chart and explain in natural language why certain features are considered abnormal.

[0158] In one embodiment of the present invention, in a network security monitoring system, a visualization report may be presented as follows: "In the past 24 hours, the two features of 'connection request frequency' and 'packet size anomaly' have contributed the most to the anomaly detection results. Specifically, 'connection request frequency' is 300% higher than the normal level, and the frequency of 'packet size anomaly' is 5 times the usual. This pattern is highly consistent with known DDoS attacks."

[0159] Overall, this feature explanation report generation method greatly improves the interpretability and credibility of anomaly detection systems. It not only helps analysts better understand and verify detection results, but also provides clear and actionable information for decision makers. In areas that require high accountability and transparency, this interpretability is particularly important because it allows the system's decision-making process to be audited and verified.

[0160] The response strategy in step 4 mainly includes the following steps:

[0161] Step 1-1, calculate the influence of feature importance:

[0162]

[0163]

[0164] in, and Respectively represent the influence of the pth and qth features, represents the model output of the feature vector x, and Respectively represent the model output after removing features p and q;

[0165] Step 1-2, abnormal overall detection abnormal degree:

[0166]

[0167] in, represents the overall abnormal threshold, represents the kth anomaly score, K is the total number of participants, I() is the indicator function, and A is the overall degree of anomaly;

[0168] Step 2-1, detect abnormal patterns:

[0169]

[0170] in, and The contribution to all features and the pth feature respectively, Represents the overall impact before selection. represents the overall impact after selection, p represents the corresponding p-th feature, K represents the number of participants, is the abnormal pattern score of the pth feature;

[0171] Step 3-1: Detect the overall abnormality and calculate the score:

[0172]

[0173] in, is the weight of the pth feature, P is the total number of features, and S is the overall abnormality score.

[0174] The process of formulating a response strategy is a key step in converting abnormal detection results into actual actions. The response strategy formulation method of the present invention takes into account information at multiple levels, thereby generating more accurate and effective response measures.

[0175] First, the system calculates the influence of feature importance. The core idea of ​​this step is to quantify the importance of each feature by comparing the difference between the original model output and the model output after removing a feature. The advantage of this method is that it can intuitively show which features have the greatest impact on the anomaly detection results, thus providing an important reference for the subsequent response strategy formulation.

[0176] For example, in a network security monitoring system, if the importance of the feature "connection request frequency" is found to have suddenly increased significantly, it may mean that the system is under a DDoS attack. This information can directly guide the formulation of response strategies, such as adding relevant protection measures or isolating the affected network area.

[0177] Secondly, the system calculates the degree of abnormality of the overall detection of anomalies. This step comprehensively considers the abnormal conditions of all participants and gives a global evaluation of the degree of abnormality. The advantage of this method is that it can give an overall evaluation of the abnormality in a distributed environment, which helps to determine whether the anomaly is a local problem or a systemic problem.

[0178] Then, the system detects abnormal patterns. This step identifies specific abnormal patterns by comparing the contribution of a certain feature with the overall contribution. This method can more finely characterize the specific manifestations of abnormalities and provide a basis for formulating targeted response strategies.

[0179] Finally, the system calculates the overall abnormality score. This score takes into account the abnormality pattern scores and importance weights of each feature and gives a comprehensive abnormality assessment. The advantage of this comprehensive scoring method is that it can balance the importance of different features and give a more objective and comprehensive abnormality assessment.

[0180] In one embodiment of the present invention, in a financial risk monitoring system, the system may formulate a response strategy as follows: when the abnormal pattern scores of the two features "frequency of large transactions" and "number of cross-border transfers" are detected to be significantly increased, the system will automatically increase the monitoring weight of these features. If the overall abnormality score exceeds a preset threshold (for example, 0.8), the system will immediately trigger a high-level alarm, freeze the relevant accounts, and notify the manual review team to conduct an in-depth investigation.

[0181] In general, this multi-level, multi-angle response strategy formulation method can generate more accurate and effective response measures based on the specific characteristics and degree of the anomaly. It is particularly suitable for complex distributed big data environments and can effectively respond to various abnormal situations while protecting normal business.

[0182] The anomaly detection enhancement strategy in step 5 is to calculate the final anomaly model selection weight through the following formula and use it to control the adaptive model parameters:

[0183]

[0184] Among them, α is the learning rate, represents the weight of the kth participant in the tth iteration, Indicates the abnormal model selection result corresponding to the kth participant, The average of the anomaly model selection results for all participants, is the weight gradient; the anomaly detection model selection includes: (1) selecting a corresponding anomaly detection model:

[0185]

[0186] in, is the selected anomaly detection model, is the prediction result of the ith model, is the input data of the kth participant, is the parameter of the i-th model;

[0187] (2) The abnormal model is the final abnormal result model, and the normal result model is the classification result model:

[0188]

[0189] in, For the final result, is the output of the abnormal result model, is the output of the common classification result model, is the predicted probability of the abnormal model, is a preset threshold. Strengthening strategy and model selection method of anomaly detection system. This is another innovation of the present invention, which enables the system to be continuously optimized and adjusted according to the actual detection effect, thereby continuously improving the accuracy and efficiency of anomaly detection.

[0190] First, the system calculates the weight of each participant through an iterative update formula. This formula takes into account the difference between the participant's current anomaly detection results and the average level, as well as the gradient information of the weight. The advantage of this method is that it can dynamically adjust the importance of each participant in the entire system based on its performance.

[0191] For example, in a multi-bank anti-money laundering system, if a bank's anomaly detection model performs particularly well and can identify suspicious transaction patterns that other banks have not discovered, the weight of this bank will be increased. This dynamic adjustment mechanism ensures that the system can maximize the use of the best performing model, thereby continuously improving the overall detection effect.

[0192] Secondly, the system uses a probability maximization method when selecting anomaly detection models. This method selects the model that can maximize the probability of predicting the result under given input data conditions. The advantage of this selection strategy is that it can select the most suitable detection model based on the data characteristics of different participants.

[0193] Finally, the system uses a threshold to decide whether to classify a sample as an anomaly. The flexibility of this method lies in the fact that the sensitivity and accuracy of detection can be balanced by adjusting the threshold τ. A lower threshold will increase the sensitivity of detection and may capture more potential anomalies, but it may also increase the false alarm rate. A higher threshold will increase the accuracy of detection, but may miss some minor anomalies.

[0194] In one embodiment of the present invention, in a large-scale network security monitoring system, the system may apply this reinforcement strategy as follows: Initially, all participating nodes have equal weights. Over time, nodes that can detect network attacks earlier and more accurately will receive higher weights. For example, if a node has successfully detected multiple zero-day attacks in the past week, while other nodes have not been able to detect them, the weight of this node will increase significantly. The system will rely more on the detection results of this node, thereby improving the overall security protection capabilities.

[0195] At the same time, the system dynamically adjusts the threshold τ for anomaly determination. During periods of frequent network attacks, the system may reduce the τ value, such as from 0.8 to 0.7, to increase the sensitivity of detection. This means that even minor anomalies will be marked, although this may increase some false positives. Conversely, during periods of relatively stable network conditions, the system may increase the τ value, such as from 0.7 to 0.85, to reduce false positives and focus only on the most significant anomalies.

[0196] This adaptive reinforcement strategy brings many advantages. First, it can fully utilize the advantages of each node in the distributed system to continuously improve the performance of the entire system. Second, it can flexibly adjust the detection strategy according to the actual situation to achieve a balance between sensitivity and accuracy. Third, this method also facilitates the introduction of new detection models into the system, and the weight of the new model in the system can be gradually adjusted according to its actual performance.

[0197] Preferably, the anomaly detection enhancement strategy of the present invention can also be combined with a time factor. For example, a time decay factor β (0 < 1), modify the weight update formula to:

[0198]

[0199] in, is the weight at time t, is the weight at the previous moment, is the time decay factor, is the learning rate, is the score of the kth sample is the average score, Update the gradients for the weights.

[0200] This allows the system to focus more on recent performance rather than overly relying on historical data. This strategy is particularly effective in fast-moving environments such as abnormal trading detection in the stock market.

[0201] In another embodiment of the present invention, in a cross-industry credit risk assessment system, this reinforcement strategy can be applied as follows: the system may initially consider credit assessment models from different participants such as banks, insurance companies, and e-commerce platforms equally. Over time, the system will gradually find that the bank's model performs better in assessing the risk of large loans, while the e-commerce platform's model is more accurate in assessing the credit of small and micro enterprises. Based on this, the system will dynamically adjust the weights of each participant's model in different scenarios, thereby achieving more accurate credit risk assessment.

[0202] In the verification process of the present invention, a typical cross-industry financial risk monitoring scenario was selected as the test background. This scenario involves multiple banks, insurance companies and Internet financial platforms, aiming to jointly identify and prevent potential financial fraud. The method of the present invention was compared with two traditional methods to fully demonstrate its superiority.

[0203] Example 1 adopts the distributed big data anomaly detection method based on federated learning proposed in the present invention. In this embodiment, each participant retains its own original data and only shares encrypted model parameters and gradient information. The system adopts dynamic feature extraction, multi-model fusion and adaptive response strategy to achieve efficient and privacy-preserving anomaly detection.

[0204] Comparative Example 1 adopts the traditional centralized data analysis method. In this method, all participants need to upload their data to a central server for unified analysis. Although this method can utilize global data, it has a serious risk of privacy leakage.

[0205] Example 2 uses a simple distributed method, where each participant independently runs its own anomaly detection model and then decides the final result through a voting mechanism. Although this method protects data privacy to a certain extent, it is difficult to fully utilize global information.

[0206] Five key indicators were selected to evaluate the performance of these methods: detection accuracy, false positive rate, false negative rate, privacy protection degree and system response time. These indicators comprehensively reflect the effectiveness, reliability and practicality of the anomaly detection system.

[0207] The detection accuracy is calculated by comparing with the manually annotated test set. The false positive rate and false negative rate respectively count the proportion of normal samples that are wrongly judged as abnormal and the proportion of real abnormal samples that are not detected. The degree of privacy protection is evaluated by simulating the success rate of attackers trying to reconstruct the original data. The system response time records the average time from receiving data to outputting the final detection result.

[0208] After three months of rigorous testing, the following results were obtained:

[0209] index Example 1 Comparative Example 1 Comparative Example 2 Detection accuracy 94.5% 92.1% 85.7% False Positive Rate 3.2% 5.8% 8.9% False negative rate 2.3% 2.1% 5.4% Privacy protection level 98.7% 45.3% 96.2% System response time 1.2s 0.8s 2.5s

[0210] From these results, we can clearly see the superiority of the method of the present invention. Example 1 is significantly better than the other two methods in detection accuracy, reaching a high level of 94.5%. This fully demonstrates the effectiveness of the dynamic feature extraction and multi-model fusion strategy proposed by the present invention. It can capture more subtle abnormal patterns and performs well in different types of financial fraud detection.

[0211] In terms of false alarm rate, Example 1 also performs best, only 3.2%. This means that the method of the present invention can greatly reduce unnecessary inspections and customer dissatisfaction caused by false alarms, and improve the credibility of the system. Although Example 1 is slightly higher than Comparative Example 1 in terms of false alarm rate, this small difference is acceptable considering the huge advantage of privacy protection.

[0212] The most striking indicator is the degree of privacy protection. Example 1 achieved an astonishing 98.7% protection rate, far exceeding the 45.3% of Comparative Example 1. This means that even when subjected to malicious attacks, the original data of the participants can be highly protected. This is extremely important for financial institutions because it not only complies with increasingly stringent data protection regulations, but also enhances customer trust.

[0213] In terms of system response time, although Example 1 is not as good as the centralized comparative example 1, it is significantly better than the fully distributed comparative example 2. Considering the high accuracy and strong privacy protection provided by Example 1, the average response time of 1.2 seconds is very ideal and can fully meet the needs of real-time monitoring.

[0214] These test results fully demonstrate that the present invention has achieved an excellent balance between accuracy, privacy protection and efficiency. It not only surpasses traditional methods in detection performance, but also sets a new standard in protecting data privacy. This method provides financial institutions with a solution that can fully utilize the value of data while strictly protecting customer privacy, and is expected to play an important role in future financial risk control.

[0215] In summary, Example 1 is undoubtedly the best example. It demonstrates the great potential of the present invention in practical applications and opens up a new path for anomaly detection in the financial industry. This method is not only applicable to the financial field, but can also be extended to other fields that require high privacy protection, such as medical care and network security, and has broad application prospects.

[0216] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Distributed big data anomaly detection system based on federated learning, It is characterized in that It includes: a dynamic feature extractor, a federated anomaly detector, a federated anomaly interpreter, a federated anomaly response coordinator, and a federated learning optimizer; the dynamic feature extractor is used to continuously monitor the real-time data stream and extract features to form a feature vector; The federated anomaly detector is used to determine whether the monitoring data stream is abnormal; The federated exception interpreter is used to quantify the importance of features and analyze the causes of exceptions; the federated exception response coordinator is used to coordinate the response strategies of corresponding participants according to the causes of the exceptions; the federated learning optimizer adjusts system parameters according to the results of the exception response strategy.

2. The anomaly detection system according to claim 1, It is characterized in that The dynamic feature extractor includes: a multi-feature extraction module, which is used to obtain real-time data of participants and extract multi-dimensional feature vectors through time series and space modeling methods; a detector selection module, which dynamically selects anomaly detectors based on dynamically changing weights and continuously optimizes detection efficiency under limited resources; the multi-feature extraction module includes a multi-scale spatiotemporal convolution model and a heterogeneous network embedding model; the multi-scale spatiotemporal convolution model collects and aggregates local spatiotemporal features in a non-overlapping manner at each scale to extract multi-granularity spatiotemporal features; by fusing the features of each sampling scale, a spatiotemporal feature vector is formed; the input of the heterogeneous network embedding is the spatiotemporal features extracted by the previous module, and by constructing a local spatiotemporal feature graph, using edge convolution to aggregate adjacent node features, and then performing feature mapping from structure to node.

3. The anomaly detection system according to claim 2, Features: The multi-feature extraction module constructs a temporal feature graph through the spatiotemporal feature vector, aggregates the features of adjacent nodes using the edge convolution model, calculates the mapping from structure to node through the self-attention model, and combines the edge convolution and self-attention layers to realize the splicing and supplementation of spatiotemporal features and heterogeneous network features to complete feature extraction; The detector selection module estimates dynamic weights based on the characteristic distribution information detector performance statistics of each participant's local data stream, and dynamically selects detectors according to the degree of influence on the anomaly detection results.

4. The anomaly detection system according to claim 3, Features: The dynamic weight is dynamically adjusted based on the influence of the anomaly detection results, and the calculation method is: ； Where j represents the serial number of the federated learning participant, d represents the number of local convolutional neural network layers in the federated learning, is the number of convolution kernel weights corresponding to the local convolutional neural network layer d of the jth participant, is the kth convolution kernel weight, is the local detector result of the jth participant, is the federated learning result, is the difference between the local detector result of the jth participant and the federated learning result; the dynamic weight of the number of local convolutional neural network layers d in the federated learning can be expressed as: ； Among them, J is the total number of participants; the selection index of the dynamic selection detector is the maximum value of the maximum chi-square correlation value between the vector and the convolution kernel, and the calculation formula is: ； in, Represents the vector corresponding to the i-th convolution kernel, K is the number of convolution kernels, is the eigenvector, F is the number of vectors, represents the chi-square correlation calculation function, Indicates the similarity between the i-th convolution kernel and the overall maximum vector, select as a selection indicator.

5. An anomaly detection method for a distributed big data anomaly detection system based on federated learning according to any one of claims 1 to 4, It is characterized in that The specific method steps include: Step 1: Dynamically extract data features of participants; extract data features of participants in real time through two levels of heterogeneous network embedding and multi-feature extraction; Step 2: Anomaly scoring and anomaly knowledge distillation; Split the federated learning model based on time series, space and overall weights, and perform abnormal threshold evaluation on the intermediate model; Strengthen according to the potential knowledge fusion method and output the final result report; Step 3: Generate feature explanatory report; Facilitate real-time updates of anomaly detection results using interpretive techniques and interpretive reports; Step 4: Set up a response strategy; Design adaptive response strategies based on dynamic feature interpretation reports; Step 5: Strengthen the anomaly detection system; Adjust system anomaly detection related parameters according to the adaptive response strategy.

6. The distributed big data anomaly detection method based on federated learning according to claim 5, It is characterized in that The method of heterogeneous network embedding and spatiotemporal feature extraction in step 1 is: Step 1) Extract continuous value features and category features from the data stream of the participants, generate a time feature vector based on the time series features, collect sampled local features at each scale, and aggregate them at multiple sampling scales in this form; aggregate features from multiple discrete scales to extract multi-granularity spatiotemporal features; the aggregation formula of the lth layer is as follows: ； Where N represents the length of the input feature vector, represents the kth time series; To extract multi-granularity spatiotemporal features; Step 2) Use multi-head attention and average attention layers to calculate the similarity of network node features; after aggregating adjacent node features through edge convolution, map from structure to node features to obtain the final node embedding; the similarity calculation formula of the average attention layer is as follows: ； Among them, W represents the input similarity matrix, Q, K, and V represent query, key, and value matrices respectively, d_k represents the number of input matrix nodes, softmax is the attention head, and S represents the output of the input through the multi-head attention mechanism; the calculation formula of the multi-head attention mechanism is: ； ； in, , and represents the attention weight matrix of different heads, and h represents the number of attention heads; Step 2-1, node neighbor representation based on attention mechanism: ； in, represents the node embedding vector of node i, represents the attention weight between nodes i and j, N(i) represents the set of adjacent nodes of node i, W is the weight matrix, For the activation function step 2-2, aggregate the adjacent node features through the edge convolution method to calculate the final node embedding: ； in, and Represent the features of nodes i and j respectively, N(i) represents the set of adjacent nodes of node i, and MLP is a multi-layer perceptron; Step 2-3: Use multi-scale spatiotemporal features as temporary embedding to construct network features; the final node embedding is and Cascade system: ； Step 2-4: Embed the node As the feature representation of the local data flow, node features are exchanged between participants and inter-node features are output.

7. The distributed big data anomaly detection method based on federated learning according to claim 5, It is characterized in that The calculation strategy of the abnormal threshold evaluation in step 2 is: obtain the abnormal detection results from the participants, set the threshold θ to sort the participants; use the intermediate model for participant selection, and input the kth participant data into the intermediate result model, and determine the abnormal score by judging the abnormal type of the intermediate result model; the abnormal evaluation score calculation formula is as follows: ； in, is the threshold corresponding to the classifier i of the intermediate result model, k represents the serial number of all participants, n is the number of participants, represents the abnormal score corresponding to the kth participant, represents the feature extractor output corresponding to the kth participant, is the i-th intermediate result model pair The output of , I() is the indicator function; The potential knowledge fusion method is used to integrate the intermediate result models of k participants, and F is used to fuse and strengthen the normal classifier and the abnormal classifier. The calculation formulas are: ； ； Among them, N and P represent the activation function normal classifier and the activation function abnormal classifier respectively, and F is the fusion function. The calculation method of fusing the intermediate result model and strengthening the anomaly detection task is expressed as follows: ； in, and The final activation functions for normal and abnormal detection are and is the weight, b is the bias term, E is the intermediate result model, and R is the final detection result; Step 2 is enhanced according to the potential knowledge fusion method, reasonably allocates the participants, and outputs the federated anomaly detection result report.

8. The distributed big data anomaly detection method based on federated learning according to claim 5, It is characterized in that The steps for generating the feature interpretation report in step 3 are as follows: Step 1: Generate a feature contribution graph using the interpretation method: Calculate the feature contribution and generate a ranking graph of the feature’s influence on the overall performance for participant selection. The formula is as follows: ； in, is the importance of feature j, is the model output for the original input x, is the model output after removing feature j; Step 2-1: Obtain the feature contribution matrix through the maximum difference method: ； in, and Represent the importance mean and standard deviation of feature i respectively, The standard Euclidean distance between the feature vector after removing variable j for training corresponding feature i and all feature vectors to be evaluated Step 2-2, further calculate through abnormal contribution to obtain the feature contribution vector: ； in, is the importance of feature i, is the mean importance value of all features in the test set, is the standard deviation; Step three generates a visualization report to facilitate the user's understanding of the decision-making process of the anomaly detection model; the visualization report includes: (1) an explanation of the steps for calculating feature contribution; and (2) an explanation of the description of the anomaly feature.

9. The distributed big data anomaly detection method based on federated learning according to claim 5, It is characterized in that The response strategy in step 4 mainly includes the following steps: Step 1-1, calculate the influence of feature importance: ； ； in, and Respectively represent the influence of the pth and qth features, represents the model output of the feature vector x, and Respectively represent the model output after removing features p and q; Step 1-2, abnormal overall detection abnormal degree: ； in, represents the overall abnormal threshold, represents the kth anomaly score, K is the total number of participants, I() is the indicator function, and A is the overall degree of anomaly; Step 2-1, detect abnormal patterns: ； in, and The contribution to all features and the pth feature respectively, Represents the overall impact before selection. represents the overall impact after selection, p represents the corresponding p-th feature, K represents the number of participants, is the abnormal pattern score of the pth feature; Step 3-1: Detect the overall abnormality and calculate the score: ； in, is the weight of the pth feature, P is the total number of features, and S is the overall abnormality score.

10. The distributed big data anomaly detection method according to claim 5, It is characterized in that The anomaly detection enhancement strategy in step 5 is to calculate the final anomaly model selection weight through the following formula, and use it to control the adaptive model parameters: ； Among them, α is the learning rate, represents the weight of the kth participant in the tth iteration, Indicates the abnormal model selection result corresponding to the kth participant, The average of the anomaly model selection results for all participants, is the weight gradient; the anomaly detection model selection includes: (1) selecting a corresponding anomaly detection model: ； in, is the selected anomaly detection model, is the prediction result of the ith model, is the input data of the kth participant, is the parameter of the i-th model; (2) the abnormal model is the final abnormal result model, and the normal result model is the classification result model.

Images