Adaptive distributed network threat detection and response system and method

Abstract

The invention discloses an adaptive distributed network threat detection and response system and method, and the system comprises a data collection module which is used for capturing network traffic in real time and converting the network traffic into structured data; the data analysis module is used for performing parallel analysis on the structured data by adopting a plurality of anomaly detection algorithms; the algorithm fusion module is used for counting the abnormal judgment times of each algorithm on the same sample through a voting mechanism, and outputting a final classification result according to a preset threshold value; the method comprises the following working steps: concurrently capturing network traffic through distributed nodes, analyzing the network traffic into structured data, operating an isolated forest algorithm, a DBSCAN algorithm and a K-means algorithm in parallel, and outputting an anomaly judgment result; and fusing multi-algorithm results and generating final classification through a voting mechanism. The invention provides an efficient, intelligent and distributed network security protection system, which can effectively detect and respond to various network threats, and is particularly suitable for large-scale distributed environments and dynamic network security defense requirements.

Description

Technical Field

[0001] The present invention relates to the field of network threat detection, and particularly to an adaptive distributed network threat detection and response system and method. Background Art

[0002] Existing network threat detection methods mostly rely on static rules and are difficult to effectively cope with constantly changing attack means. In addition, traditional security measures often require manual intervention, resulting in a long response time and being unable to promptly prevent potential threats. Therefore, there is an urgent need for a network threat detection solution that can adaptively learn and adjust to improve network security protection capabilities.

[0003] As described above, for this reason, we have designed an adaptive distributed network threat detection and response system and method to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to solve the deficiencies existing in the prior art and propose an adaptive distributed network threat detection and response system and method.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: An adaptive distributed network threat detection and response system, comprising: A data collection module, configured to capture network traffic in real time and convert it into structured data; A data analysis module, which uses several anomaly detection algorithms to perform parallel analysis on the structured data; An algorithm fusion module, which counts the number of anomaly determinations of the anomaly detection algorithms for the same sample through a voting mechanism and outputs a final classification result according to a preset threshold; An adaptive policy adjustment module, based on the -learning algorithm of reinforcement learning, dynamically optimizes security policies, including defining a state space, an action space, and a reward function, and realizes policy iteration through table update; A response processing module, which calls a network management system through a RESTful API with permission verification to perform isolation or recovery operations.

[0006] Preferably, the data collection module specifically includes: Listening on a specified network interface through the gopacket library or libpcap library of the Go language; Parsing the data packet to each layer of TCP / IP, extracting source / destination IP, port number, and protocol type fields and storing them as a structure; The data collection module performs concurrent processing of multiple data packets based on goroutines and transmits the data packets to the central analysis node through the Kafka message queue.

[0007] Preferably, the several anomaly detection algorithms include one or more of the Isolation Forest algorithm, DBSCAN, and K-means algorithm, and the data analysis module executes the following algorithms: Isolation Forest algorithm: Randomly divide the samples into isolation trees, calculate the path length to generate an anomaly score, and the formula is: ; where is the average path length of the sample in all isolation trees; is the normalization factor for a given number of samples , and the calculation formula is: ; In the formula, is the harmonic number DBSCAN: Based on density clustering, mark the noise points as anomalies, and the condition for defining the noise points is: ; where represents the number of neighbors of point p within the neighborhood radius ; is the preset minimum number of neighbor threshold; K-means algorithm: Determine anomalies by the distance threshold between the sample and the cluster centroid. The goal of K-means is to minimize the within-cluster sum of squares error: ; where SEE represents the within-cluster sum of squares error; represents a data point in the cluster ; represents the total number of clusters divided in the clustering algorithm; is the th cluster; is the centroid of the cluster , and the update formula is: .

[0008] Preferably, the voting mechanism of the algorithm fusion module includes: Set a threshold T. If the number of algorithms that determine a sample as abnormal ≥ T, then mark it as finally abnormal; The weights of each algorithm are adjustable, and weighted voting is supported.

[0009] Preferably, in the adaptive policy adjustment module: The state space includes the real-time attack frequency, detection rate, and false alarm rate; The action space includes enabling new detection rules and adjusting resource allocation; The reward function is: R = w1 × detection rate - w2 × false alarm rate - w3 × response time, where w1, w2, and w3 are adjustable weights.

[0010] Preferably, the response processing module includes: Role-based permission management to restrict API operation permissions; Automated scripts regularly scan infected devices and trigger repair processes.

[0011] Preferably, the system performs high-concurrency processing through the Go language and supports integration with third-party services such as Kafka and Nginx.

[0012] A method for an adaptive distributed network threat detection and response system includes the following working steps: Step S1: Use the data collection module to concurrently capture network traffic through distributed nodes and parse it into structured data; run the Isolation Forest algorithm, DBSCAN, and K-means algorithms in parallel through the data analysis module, and output the abnormal determination result; Step S2: Based on the algorithm fusion module, fuse the results of multiple algorithms and generate a final classification through a voting mechanism; Step S3: Use the adaptive policy adjustment module, based on the -learning algorithm of reinforcement learning, dynamically adjust the security policy and update the table, -update rule of the learning algorithm: ; Among them, represents the current state, represents the action taken, represents the reward, is the learning rate, is the discount factor, and respectively represent the next state and action of the current state; Step S4: Call the API for permission verification through the response processing module to perform isolation or recovery operations.

[0013] Preferably, the dynamic adjustment security policy includes: Adopt the ε-greedy strategy to balance exploration and exploitation; Optimize the table through the reward function R = w1 × detection rate - w2 × false alarm rate - w3 × response time. Table.

[0014] Preferably, the method is deployed in a distributed architecture, including: Multi-node data collection and load balancing; Redundant design of key components and master-slave switchover fault tolerance mechanism.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. High efficient concurrent processing ability: Using the Go language enables the system to efficiently handle high-concurrency tasks, especially suitable for data collection modules that need to listen to multiple network interfaces simultaneously. This not only improves the speed of data capture but also ensures the real-time performance and response speed of the system.

[0016] 2. Intelligent anomaly detection algorithm: The data analysis module adopts a variety of advanced anomaly detection algorithms, such as IsolationForest, DBSCAN, and K-means, etc. These algorithms can effectively identify complex network attack patterns and improve the accuracy and robustness of anomaly detection by integrating the results of different algorithms through a voting mechanism.

[0017] 3. Dynamically adjustable security policy: The adaptive policy adjustment module uses reinforcement learning algorithms (such as -learning) to dynamically adjust the security policy according to real-time feedback. This adaptive mechanism enables the system to quickly respond to new threat types and changing attack means, enhancing the overall network security protection ability.

[0018] 4. Highly flexible and scalable architecture design: The system adopts a typical distributed architecture design, supporting multi-node data collection, distributed execution of algorithms and policies, as well as load balancing and fault tolerance mechanisms. This architecture not only improves the stability and reliability of the system but also facilitates future expansion and upgrade to meet the growing network security requirements.

[0019] 5. Automated operation and management: The response processing module interacts with the network management system through the RESTful API to perform isolation and recovery operations. In addition, automated scripts are written to regularly scan and repair infected devices, greatly reducing the need for manual intervention and improving the operation and maintenance efficiency.

[0020] 6. Enhanced User Experience and Security: The permission management mechanism ensures the security of API calls and prevents unauthorized operations. Through strict permission control, the system is protected from potential security threats, while also providing a convenient service experience for legitimate users.

[0021] 7. Improvement in Resource Utilization Efficiency: The load balancing mechanism and redundant design ensure the high availability and stability of the system. Key components such as databases and message queues adopt a master-slave architecture, which can quickly switch to the standby node in case of a failure of the master node, ensuring the continuity of services and avoiding the risks brought by single-point failures. Brief Description of the Drawings

[0022] Figure 1 It is a module composition diagram of an adaptive distributed network threat detection and response system proposed by the present invention; Figure 2 It is a schematic diagram of the functional composition of the data collection module in an adaptive distributed network threat detection and response system proposed by the present invention; Figure 3 It is a schematic diagram of the functional composition of the data analysis module in an adaptive distributed network threat detection and response system proposed by the present invention; Figure 4 It is a schematic diagram of the functional composition of the adaptive policy adjustment module in an adaptive distributed network threat detection and response system proposed by the present invention. Detailed Embodiments

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0024] Referring to Figures 1 - 4 , the present invention proposes an adaptive distributed network threat detection and response system and method. The system aims to address the deficiencies in existing network security measures, especially those methods that rely on static rules and require manual intervention, which are difficult to effectively respond to constantly changing attack means and have a long response time. By using the Go language, the system can efficiently handle high-concurrency tasks by leveraging its lightweight thread characteristics, which is particularly important for the data collection module that listens to data from multiple network interfaces simultaneously.

[0025] The core functions of the system include multi-node data collection, distributed execution of algorithms and strategies, load balancing, and fault tolerance mechanisms. Specifically, the data collection module can capture and convert network traffic into a structured format in real time for subsequent analysis; the data analysis module applies anomaly detection algorithms such as IsolationForest, DBSCAN, and K-means algorithms for anomaly detection, and fuses the results of different algorithms through a voting mechanism to determine the final classification; the adaptive policy adjustment module uses reinforcement learning algorithms (such as -learning algorithm) to dynamically adjust security policies, improving the system's response ability and protection effect.

[0026] In addition, the system also designs a simulation environment to test different policies and selects the best policy for application in the real environment according to the evaluation results. The entire system architecture reflects a high degree of flexibility, scalability, and stability, providing an effective solution for modern network security.

[0027] The adaptive distributed network threat detection and response system implemented based on the Go language proposed by the present invention. The Go language supports lightweight threads, making high-concurrency task processing simple and efficient, which is crucial for the data collection module that needs to listen to multiple network interfaces simultaneously. The system proposed by the present invention is a typical distributed architecture, which is reflected in the following aspects: Multi-node data collection: Through data collection modules deployed in different geographical locations or network regions, comprehensive monitoring of the entire network environment can be achieved.

[0028] Distributed execution of algorithms and strategies: The data analysis module and the adaptive policy adjustment module can run in parallel on multiple computing nodes, accelerating the anomaly detection process and dynamically adjusting security policies according to real-time feedback.

[0029] Load balancing and fault tolerance mechanisms: Services built using the Go language can achieve load balancing through simple configuration, ensuring the stability and reliability of the system; at the same time, corresponding countermeasures are available for any single point of failure.

[0030] I. Data collection module

[0031] 1. Real-time capture

[0032] Real-time capture of network traffic is carried out by using the gopacket library or libpcap library of the Go language. These libraries allow developers to listen to specific network interfaces and process the received data packets with high efficiency. The specific steps are as follows: (1) Initialize listening: Select and initialize the network interface to be listened to.

[0033] (2)Capture Packets: Use the gopacket library or the libpcap library to capture packets in network traffic.

[0034] (3)Concurrent Processing: Achieve efficient concurrent processing through Go language goroutines to ensure high throughput.

[0035] 2. Data Conversion The captured raw data is decomposed into parts such as protocol headers and payloads by a parser, and then the information is organized into structured data using structures. The specific steps are as follows: (1)Parsing Process: Split the packet according to different layers of the TCP / IP model (physical layer, data link layer, network layer, transport layer, and application layer); (2)Extract Key Information in the Packet: Such as source IP address, destination IP address, port number, protocol type, etc., for subsequent analysis; (3)Structured Storage: Store the parsed and extracted packet information in a structure to form structured data for use by the data analysis module.

[0036] II. Data Analysis Module

[0037] Adopt various anomaly detection algorithms such as the IsolationForest algorithm, DBSCAN, and K-means algorithm for anomaly detection.

[0038] 1. IsolationForest Algorithm

[0039] (1)Construct a Tree Structure: Randomly select samples from the dataset (structured data) and randomly select a feature as the splitting point, and recursively divide the dataset (structured data) into subsets until all samples are isolated.

[0040] (2)Calculate the Path Length: For each sample, record the average path length required to reach the leaf node in each tree. Outliers usually require a shorter path to be isolated.

[0041] (3)Anomaly Scoring: The anomaly score is based on the average path length required for a sample to reach the leaf node in each tree. Outliers usually require a shorter path to be isolated, so the shorter the path length, the higher the anomaly score.

[0042] (4)The calculation formula is ; where is the sample in the average path length of all isolation trees; For a given number of samples The normalization factor is calculated as: ; ( is the harmonic number).

[0043] 2. DBSCAN (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) (1) Define core points: If a point is within its radius If there are at least MinPts points in the neighborhood of , then the point is a core point. The conditions for defining noise points are: ; in, Indicates that point p is in the neighborhood radius The number of neighbors within is the preset minimum neighbor point number threshold.

[0044] (2) Cluster formation: Starting from any unvisited core point, find all the core points directly or indirectly connected to it to form a cluster.

[0045] (3) Marking noise points: Points that do not belong to any cluster are considered noise points.

[0046] 3. K-means algorithm (1) Initialize the centroid: Randomly select several points as the initial centroid.

[0047] (2) Assign clusters: Assign each data point to the cluster with the closest centroid.

[0048] (3) Update the centroid: Recalculate the centroid of each cluster.

[0049] (4) Iterative optimization: Repeat the above two steps until the centroid no longer changes or the maximum number of iterations is reached.

[0050] (5) If the distance from a data point to the centroid of its cluster is greater than the threshold T, the data point is considered to be an outlier.

[0051] The coordinates of the centroid are the average of the coordinates of all the data points in the cluster along that dimension. For each dimension within the cluster, add up the coordinates of all the data points along that dimension. Divide the sum by the number of data points in the cluster to get the mean for that dimension. Combine the means of all dimensions to form the new centroid coordinates.

[0052] The goal of K-means is to minimize the intra-cluster squared error: ; in, SEE stands for within-cluster squared error; Representation Cluster A data point in Indicates the total number of clusters divided in the clustering algorithm; For the clusters; Cluster The update formula of the centroid is: .

[0053] 3. Algorithm Fusion Module In order to integrate the results of different algorithms, a voting mechanism is used to determine the final classification result. The specific implementation steps are as follows: (1) Single algorithm output: Each algorithm performs anomaly detection on each sample and gives a judgment on whether it is abnormal; (2) Counting the number of anomalies: For each sample, count the number of times it is judged as an anomaly by each algorithm; (3) Decision rules: Set a threshold (for example, if the majority of algorithms consider the sample to be abnormal, it will be marked as an abnormality). If the majority of algorithms consider the sample to be abnormal, it will eventually be marked as an abnormality. The Isolation Forest algorithm makes a decision based on the abnormality score being lower than the standard value; DBSCAN makes a decision based on marking it as a noise point; and the K-means algorithm makes a decision based on the distance from the data point to the centroid of the cluster to which it belongs being greater than the threshold.

[0054] 4. Adaptive Strategy Adjustment Module Reinforcement Learning Algorithms ( -learning algorithm) The basic idea of ​​reinforcement learning is that the agent learns the best behavior strategy by interacting with the environment. -learning algorithm is a model-free reinforcement learning algorithm. -The update rule of the learning algorithm is: ; in, Indicates the current state. Indicates the action taken, Indicates reward, is the learning rate, is the discount factor, and respectively represent the next state and action of the current state.

[0055] Among them, : represents executing the action under the current state of value, that is, the expected cumulative reward of the current state-action pair.

[0056] : the next state in, select the action that can make value maximized corresponding to value. Its essence is to predict the best action value of the next state.

[0057] Its implementation process is as follows: (1) State space and action space: Define the state of the system (such as the current security status, attack frequency, type distribution, etc.) and the possible actions (such as adjusting resource allocation, enabling new detection rules, etc.).

[0058] (2) Table update: After the agent executes an action in each state, update the table (state-action value function) according to the reward function. The reward function can be designed according to system performance metrics (such as detection rate, false positive rate). Common metrics include: Detection rate: The proportion of successfully identifying threats.

[0059] False positive rate: The proportion of mislabeling normal traffic as threats.

[0060] Response time: The time from detecting a threat to taking countermeasures.

[0061] The reward function can be designed as: ; where w1, w2, w3 are weight parameters (adjustable weights) and can be adjusted according to specific requirements.

[0062] (3) Policy update: Select the optimal policy according to the table to guide how the system can more effectively allocate resources to cope with new threats.

[0063] Exploration and exploitation: Balance exploring new policies and exploiting existing best policies through the ε-greedy policy.

[0064] Select the optimal action: Use the greedy policy to select the action with the largest value in the current state.

[0065] Strategy Execution and Adjustment: Execute the selected actions and update the table according to the feedback, continuously optimizing the strategy.

[0066] It should be noted that: I. To achieve comprehensive monitoring of the entire network environment, multiple data collection nodes are deployed in different geographical locations or network regions. Each node is responsible for listening to the network traffic within a specific range to ensure no dead corners. The implementation details are as follows: 1. Configuration Management: Each node has its own configuration file to specify the network interface to be listened to.

[0067] 2. Concurrent Processing: Utilize the goroutines in Go language to achieve efficient concurrent processing and ensure high throughput.

[0068] 3. Data Transmission: Use a message queue (such as Kafka) to transmit the captured data packets (completed by the data collection module and captured through the gopacket library) to the central analysis node for analysis by the data analysis module.

[0069] II. Load Balancing and Fault Tolerance Mechanism Services built with Go language are easy to configure for load balancing, ensuring the stability and reliability of the system. At the same time, the system design considers the possibility of single point of failure and adopts redundant design and automatic switching measures. The implementation details are as follows: 1. Load Balancing: Use Nginx or HAProxy as a reverse proxy server to achieve load balancing of requests.

[0070] The core goal of load balancing is to reasonably distribute the client requests to multiple backend servers, thus avoiding overloading of a single server and improving the stability and reliability of the system. The following are the specific methods to achieve load balancing: The reverse proxy server (such as Nginx or HAProxy) acts as an intermediary between the client and the backend servers. It receives the client requests and distributes the requests to the backend servers according to predefined rules.

[0071] The load balancer regularly checks the health status of the backend servers. If a server is unavailable (such as response timeout or return of an error status code), the load balancer will temporarily remove it from the service pool until it resumes normal.

[0072] 2. Fault Tolerance Mechanism: Ensure the high availability of the service through heartbeat detection and automatic restart mechanism.

[0073] The working principle of heartbeat detection Heartbeat detection is a periodic communication mechanism used to confirm whether a service is running properly. Each component in the system, such as a load balancer and service nodes, will regularly send "heartbeat packets" (usually small data packets) to indicate that they are still online. If no heartbeat response is received within the specified time, the service is considered likely to have failed. For example, the load balancer may attempt to resend requests or mark the service as unavailable.

[0074] Working principle of the automatic restart mechanism The system continuously monitors the running status of critical services. If a service stops unexpectedly (such as crashing or becoming unresponsive), the monitoring system will detect it immediately. When a service failure is detected, the system will attempt to automatically restart the service.

[0075] 3. Redundant design: Key components (such as databases and message queues) adopt a master-slave architecture to ensure a quick switch to the standby node in case of a failure of the master node.

[0076] III. Data collection module 1. Hardware requirements: At least one server or virtual machine is required to run the data collection module. It is recommended to use a multi-core processor to support concurrent processing.

[0077] 2. Software dependencies: Install the Go language environment (version 1.16 or higher) and ensure that the gopacket library is available.

[0078] 3. Network interface selection: Select a suitable network interface for listening according to the actual network topology, such as eth0.

[0079] 4. Configuration file: A JSON-format configuration file can be created to manage the listening interfaces.

[0080] IV. Data analysis module 1. Data preprocessing: The captured data packets need to be parsed into structured data first and then input into the anomaly detection algorithm.

[0081] 2. Algorithm selection: Combine algorithms such as IsolationForest, DBSCAN, and K-means, and use a voting mechanism to comprehensively determine whether a data packet is abnormal.

[0082] Specific implementation method of judgment: Use the IsolationForest algorithm to perform anomaly detection on the data, record the anomaly score of each sample, and determine whether it is abnormal based on the score.

[0083] Use DBSCAN to cluster the data and mark the noise points as abnormal.

[0084] Use the K-means algorithm to cluster the data, calculate the distance from each sample to the cluster center, and determine whether it is abnormal based on the distance.

[0085] 5. Algorithm Fusion Module 1. Decision rules: Set a threshold, for example, if the majority of algorithms consider a sample to be abnormal, mark it as abnormal.

[0086] 2. Result integration: Summarize the results of different algorithms and decide the final classification through a voting mechanism.

[0087] 6. Adaptive Strategy Adjustment Module Reinforcement Learning Algorithms ( -learning algorithm) (1) Create a surface( -table), which records the initial state of each state-action pair The initial value is usually set to 0 or a random decimal value.

[0088] (2) At each step, the agent needs to decide which action to take. This can be achieved using an ε-greedy strategy.

[0089] (3) The agent executes the selected action and observes the next state returned by the environment.

[0090] (4) Use -Learning algorithm updates the formula to update the current state-action pair value.

[0091] (5) The agent continues to interact with the environment and continuously updates The table is repeated until a preset termination condition (such as a maximum number of iterations or convergence) is reached.

[0092] 7. Response Processing Module 1. API call: Interact with the network management system through RESTful API to perform isolation and recovery operations.

[0093] Use RESTful API to send an isolation request to the network management system. The API request usually includes the following information: Target ID: The object that needs to be isolated (such as IP address, MAC address, device ID, etc.).

[0094] Isolation Type: Specify the isolation method (such as blocking traffic, disabling ports, limiting access, etc.).

[0095] Duration: The time range of isolation (such as permanent isolation or temporary isolation). The above response module can be used to isolate and restore the system of abnormal network attacks detected and identified, which is a continuation of the above.

[0096] Send restore command: Use RESTful API to send a recovery request to the network management system. The API request usually includes the following information: Target ID: The object that needs to be restored (such as IP address, device ID, etc.).

[0097] Restoration Type: Specify the restoration method (such as unblocking traffic, enabling ports, restoring access rights, etc.).

[0098] 2. Permission management: ensure the security of API calls and prevent unauthorized operations.

[0099] (1) Role definition: Define different user roles (such as administrator, operator, read-only user, etc.) and assign specific permissions to each role.

[0100] Administrator: Can perform quarantine and recovery operations.

[0101] Operator: Can only view the status but cannot perform operations.

[0102] Read-only user: can only query information.

[0103] (2) Authority verification: When processing a request, the API server will check whether the caller's role and permissions meet the requirements. For example: If the caller attempts to perform an isolated operation but their role is "Read-Only User", an insufficient permissions error is returned.

[0104] This embodiment can achieve efficient concurrent processing capabilities: the use of Go language enables the system to efficiently handle high-concurrency tasks, which is particularly suitable for data collection modules that need to monitor multiple network interfaces at the same time. This not only improves the speed of data capture, but also ensures the real-time and response speed of the system. The data analysis module uses a variety of advanced anomaly detection algorithms, such as Isolation Forest, DBSCAN, and K-means, which can effectively identify complex network attack patterns and integrate the results of different algorithms through a voting mechanism, thereby improving the accuracy and robustness of anomaly detection. The adaptive strategy adjustment module uses reinforcement learning algorithms (such as -learning), dynamically adjusts the security policy according to real-time feedback. This adaptive mechanism enables the system to quickly respond to new threat types and changing attack methods, enhancing the overall network security protection ability. The system adopts a typical distributed architecture design, supporting multi-node data collection, distributed execution of algorithms and policies, as well as load balancing and fault tolerance mechanisms. This architecture not only improves the stability and reliability of the system but also facilitates future expansion and upgrade to meet the growing network security requirements.

[0105] Among them, the distributed architecture design includes: Multi-node data collection: By deploying data collection modules in different geographical locations or network regions, comprehensive monitoring of the entire network environment can be achieved.

[0106] The data analysis module and the adaptive policy adjustment module can run in parallel on multiple computing nodes, accelerating the anomaly detection process and dynamically adjusting the security policy according to real-time feedback. The response processing module interacts with the network management system through RESTful API to perform isolation and recovery operations. In addition, automated scripts are written to regularly scan and repair infected devices, greatly reducing the need for manual intervention and improving the operation and maintenance efficiency.

[0107] The permission management mechanism ensures the security of API calls and prevents unauthorized operations. Through strict permission control, the system is protected from potential security threats, while also providing a convenient service experience for legitimate users. The load balancing mechanism and redundant design ensure the high availability and stability of the system. Key components such as the database and message queue adopt a master-slave architecture, which can quickly switch to the standby node in case of a master node failure, ensuring the continuity of services and avoiding the risks brought by single-point failures.

[0108] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. An adaptive distributed network threat detection and response system, characterized in that, It includes: A data collection module for capturing network traffic in real time and converting it into structured data; A data analysis module for parallel analysis of the structured data using anomaly detection algorithms; An algorithm fusion module for counting the number of anomaly judgments of the anomaly detection algorithms for the same sample through a voting mechanism and outputting a final classification result according to a preset threshold; Adaptive policy adjustment module, based on reinforcement learning -learning algorithm, dynamically optimize the security policy, including defining the state space, action space and reward function, and perform policy iteration through table update; A response processing module for calling a network management system to perform isolation or recovery operations through a RESTful API with permission verification.

2. An adaptive distributed network threat detection and response system according to claim 1, characterized in that, The data collection module includes: Listening to a specified network interface through the gopacket library or libpcap library of the Go language; Parsing the data packet to each layer of TCP / IP, extracting source / destination IP, port number, and protocol type fields and storing them as a structure; The data collection module performs concurrent processing of multiple data packets based on goroutines and transmits the data packets to the central analysis node through the Kafka message queue.

3. An adaptive distributed network threat detection and response system according to claim 1, characterized in that The anomaly detection algorithms include one or more of the Isolation Forest algorithm, DBSCAN, and K-means algorithm, and the data analysis module executes the following algorithms: Isolation Forest algorithm: Randomly divide the samples into isolation trees, calculate the path length to generate an anomaly score, and the formula is: ； Where, For the sample Average path length among all isolated trees; For a given number of samples The normalization factor at this time is calculated by the formula: ； wherein, is a harmonic number; DBSCAN: Mark the noise points as anomalies based on density clustering, and define the conditions for noise points as: ； Where, Denote the number of neighboring points of point p within the neighborhood radius ; is a preset minimum adjacent point number threshold; K-means algorithm: Determine anomalies through the distance threshold between the sample and the cluster centroid. The goal of K-means is to minimize the within-cluster sum of squares error: ； Where, SEE represents the within-cluster sum of squares error; Indicates a data point in the cluster; Indicates the total number of clusters partitioned in the clustering algorithm; is the th cluster; For the centroid of the cluster the update formula is: .

4. An adaptive distributed network threat detection and response system according to claim 1, characterized in that, The voting mechanism of the algorithm fusion module includes: Set a threshold T. If the number of algorithms that determine a sample as an anomaly is ≥T, then mark it as a final anomaly.

5. An adaptive distributed network threat detection and response system according to claim 1, characterized in that, In the adaptive strategy adjustment module: The state space includes real-time attack frequency, detection rate, and false alarm rate; The action space includes enabling new detection rules and adjusting resource allocation; The reward function is: R = w1×detection rate - w2×false alarm rate - w3×response time, where w1, w2, and w3 are adjustable weights.

6. An adaptive distributed network threat detection and response system according to claim 1, characterized in that, The response processing module includes: Role-based permission management to restrict API operation permissions; Automated scripts regularly scan infected devices and trigger the repair process.

7. An adaptive distributed network threat detection and response system according to claim 1, characterized in that, The system performs high-concurrency processing through the Go language and supports integration with third-party services such as Kafka and Nginx.

8. A method for an adaptive distributed network threat detection and response system according to any one of claims 1-7, characterized in that, It includes the following working steps: Step S1: Use the data collection module to concurrently capture network traffic through distributed nodes and parse it into structured data; run the Isolation Forest algorithm, DBSCAN, and K-means algorithm in parallel through the data analysis module, and output the anomaly judgment results; Step S2: Based on the algorithm fusion module, fuse the results of multiple algorithms and generate a final classification through the voting mechanism; Step S3: Use the adaptive policy adjustment module to dynamically adjust the security policy based on the -learning algorithm and update the table, -update rule of the learning algorithm: ； Among them, represents the current state, represents the action taken, represents the reward, is the learning rate, is the discount factor, and respectively represent the next state and action of the current state; Step S4: Call the API with permission verification through the response processing module to perform isolation or recovery operations.

9. The method of the adaptive distributed network threat detection and response system according to claim 8, wherein The dynamic adjustment of the security policy includes: Adopting the ε-greedy strategy to balance exploration and exploitation; Optimize through the reward function R = w1 × detection rate - w2 × false alarm rate - w3 × response time Table 10. The method of the adaptive distributed network threat detection and response system according to claim 8, characterized in that, The method is deployed in a distributed architecture, including: Multi-node data collection and load balancing; Redundant design of key components and master-slave switchover fault tolerance mechanism.

Images