Dify agent-based network flow and intranet terrain analysis method and system

Abstract

The invention discloses a Dify agent-based network traffic and intranet terrain analysis method and system. The method comprises six main steps of data acquisition, data processing, module analysis, automatic response, training optimization and result output. By integrating multi-source data and utilizing advanced algorithms such as Isolation Forest, XGBoost and GNN to carry out flow analysis and terrain analysis, the method can process data in real time and update a topological graph, and multi-algorithm collaborative analysis is realized. In addition, the deployment process is simplified through the low-code workflow of the Dify platform, and the method has remarkable real-time performance, expandability and adaptivity. The implementation of the application can significantly improve the breadth and depth of data utilization, enhance the comprehensiveness and accuracy of threat detection, and provide powerful support for network security management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to a network traffic and internal network terrain analysis method and system based on a Dify agent. BACKGROUND

[0002] In the field of network security, real-time analysis of network traffic and dynamic analysis of internal network terrain are core tasks for detecting threats and maintaining network security. Existing technologies face many challenges. On the one hand, efficient processing of large-scale network traffic data is crucial, and traditional tools have deficiencies in real-time performance, making it difficult to meet the needs of dynamic network environments. On the other hand, internal network devices and connections change frequently, and static terrain analysis cannot reflect network dynamics in real time.

[0003] With the increasing complexity of network attacks, intrusion detection systems (IDS, such as Snort), security information and event management (SIEM, such as Splunk) systems, and network monitoring tools have become mainstream solutions for network security. These tools detect anomalies through rule matching or statistical analysis, for example, IDS uses signature detection to detect known threats, and SIEM analyzes logs to discover potential risks. Internal network terrain analysis usually relies on scanning tools (such as Nmap) or configuration management databases (CMDB), but generates static network views.

[0004] The closest prior art is network security solutions based on machine learning, such as Darktrace's anomaly detection system. These systems use AI to analyze traffic patterns, but usually focus on a single task (traffic analysis or threat detection), lacking dynamic integration of internal network terrain analysis. In addition, existing AI solutions are complex to deploy and difficult to quickly adapt to the needs of small and medium-sized enterprises. SUMMARY

[0005] Based on this, the embodiments of the present application provide a network traffic and internal network terrain analysis method and system based on a Dify agent. The present application integrates traffic analysis and terrain analysis through the agent of the Dify platform, providing a low-threshold, high-real-time network security solution aimed at reducing false positives, improving threat response speed, and enhancing network visibility.

[0006] In a first aspect, a network traffic and internal network terrain analysis method based on a Dify agent is provided, which includes:

[0007] Collecting real-time and historical data from network devices, logs, and configurations; wherein the data includes network traffic logs, system logs, configuration files, and real-time network data;

[0008] The collected data is cleaned, standardized and feature extracted; which specifically includes removing duplicate records, filling missing values, converting different formats of logs into a unified structure, and extracting flow rate, connection frequency, device role and log keyword key features;

[0009] The processed data is executed for traffic analysis, terrain analysis and anomaly detection to identify abnormal behavior and potential threats in the network; which uses Isolation Forest algorithm to detect abnormal traffic, XGBoost algorithm to classify known threats, GNN to construct network topology, and uses K-Means clustering and ARIMA model to identify unknown anomalies and traffic mutations;

[0010] According to the analysis and detection results, the response actions are triggered; which at least includes isolating abnormal devices, blocking malicious IP and notifying administrators;

[0011] According to the analysis and detection results, risk scores and topology graphs are generated; which includes assigning grades to generated statistics and calculating risk scores by weighting, and generating network topology graphs.

[0012] Optionally, after executing traffic analysis, terrain analysis and anomaly detection on the processed data, the method further comprises:

[0013] The agent model is trained, and the model parameters are optimized online to adapt to new threats and changes in network environment, which includes training Isolation Forest and XGBoost models using public data sets, optimizing parameters of GNN, and updating models through incremental learning, Dify feedback loop node records false positives / false negatives, and adjusts model weights.

[0014] Optionally, collecting real-time and historical data from network devices, logs and configurations includes:

[0015] Real-time and historical data is collected by configuring data collection nodes in Dify workflow, and data is obtained through API or log collection tools; wherein data collection further includes configuring data collection nodes to obtain data from multiple network sources, conducting data collection frequency statistics and conducting data source coverage statistics.

[0016] Optionally, the collected data is cleaned, standardized and feature extracted through Dify, specifically including:

[0017] Data cleaning is performed using Python tools, including removing duplicate records and filling missing values;

[0018] Log text is parsed using NLP nodes to extract event keywords;

[0019] Convert logs of different formats into a unified structure;

[0020] Extract traffic rate, connection frequency, device role, and log keyword features;

[0021] Generate statistics, including traffic rate, connection frequency, log event count, and device activity.

[0022] Optionally, perform traffic analysis, terrain analysis, and anomaly detection on the processed data, including:

[0023] Detect abnormal traffic using the Isolation Forest algorithm and generate scores based on anomaly probability;

[0024] Use the XGBoost model with a pre-trained model, input traffic features, and output threat labels;

[0025] Analyze device and connection relationships through the GNN model and calculate node importance using the PageRank algorithm;

[0026] Identify unknown anomalies through K-Means clustering and detect traffic time series mutations using the ARIMA model;

[0027] Receive processed data, perform parallel analysis, and output statistics, including abnormal traffic scores, threat type labels, topology change frequency, and key node importance.

[0028] Optionally, trigger response actions based on analysis and detection results, including:

[0029] Call firewall API to isolate abnormal devices and update access control lists to block malicious IPs;

[0030] Send notifications to administrators through Dify's Webhook node;

[0031] Configure automation nodes and integrate external tools for automated responses;

[0032] Trigger response actions based on analysis results.

[0033] In the second aspect, a network traffic and internal terrain analysis system based on Dify agents is provided, which includes:

[0034] A data collection module for collecting real-time and historical data from network devices, logs, and configurations; the data includes network traffic logs, system logs, configuration files, and real-time network data;

[0035] The data processing module is used to clean, standardize, and extract features from the collected data. Specifically, this includes removing duplicate records, filling in missing values, converting logs of different formats into a unified structure, and extracting key features such as traffic rate, connection frequency, device role, and log keywords.

[0036] The analysis module performs traffic parsing, topography analysis, and anomaly detection on the processed data to identify anomalous behavior and potential threats in the network. Specifically, it uses the Isolation Forest algorithm to detect anomalous traffic, the XGBoost algorithm to classify known threats, GNN to construct network topology, and K-Means clustering and ARIMA model to identify unknown anomalies and traffic mutations.

[0037] An automated response module is used to trigger response actions based on the analysis and detection results; wherein, the response actions include at least isolating abnormal devices, blocking malicious IPs, and notifying the administrator;

[0038] The output results module is used to generate risk scores and topology maps based on the analysis and detection results; this includes assigning levels to the generated statistical items and calculating risk scores with weights, as well as generating network topology maps.

[0039] Optionally, the system further includes:

[0040] The training and optimization module is used to train the agent model and optimize the model parameters online to adapt to new threats and changes in the network environment. The training and optimization include training Isolation Forest and XGBoost models using public datasets, optimizing the parameters of GNN, updating the model through incremental learning, and Dify's feedback loop node recording false positives / false negatives and adjusting model weights.

[0041] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the methods described in the first aspect above.

[0042] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described in the first aspect above.

[0043] The application significantly improves the breadth and depth of data utilization, and enhances the comprehensiveness and accuracy of threat detection on the basis of traditional network security analysis. First, the application goes beyond the limitations of traditional single traffic analysis, integrates multi-source data including network traffic logs, system logs, device configuration files and real-time network data, expands the dimension of data analysis, and provides more comprehensive network security insights. Secondly, for various types of data, key fields are deeply mined, and statistical items are generated through feature extraction, anomaly detection and topology analysis, improving the depth of data utilization. For example, traffic analysis uses IsolationForest and XGBoost models to accurately identify anomalies, and terrain analysis dynamically constructs network topology through GNN models. In addition, the application establishes a multi-algorithm collaborative analysis framework, combines K-Means, ARIMA and other models, proposes a statistical item level assignment and weighted calculation method, and forms a reference risk scoring system. The low-code workflow of Dify platform further reduces the deployment threshold, making the application have significant advantages in real-time, scalability and adaptability, and is suitable for various network environments. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be obtained from the provided drawings without creative labor.

[0045] Figure 1 A network traffic and internal network terrain analysis method flowchart based on a Dify intelligent agent is provided for the embodiments of the application.

[0046] Figure 2 An analysis module processing flowchart is provided for the embodiments of the application.

[0047] Figure 3 A schematic diagram of an electronic device is provided for the embodiments of the application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the application more clear, the following will further describe the application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0049] In the description of the present application, the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units is not necessarily limited to those explicitly listed steps or units, but can also include other steps or units that are not explicitly listed but are inherent to such processes, methods, products or devices, or steps or units added based on further optimization of the concept of the present application.

[0050] In the field of network security, real-time analysis of network traffic and dynamic analysis of internal network terrain are core tasks for detecting threats and maintaining network security. However, existing technologies face the following problems:

[0051] 1. Data processing bottleneck: Large-scale network traffic data needs efficient processing, and traditional tools are insufficient in real-time performance.

[0052] 2. Dynamic network changes: Internal network devices and connections change frequently, and static terrain analysis cannot meet real-time needs.

[0053] 3. Complex threat detection: Zero-day attacks and advanced persistent threats (APTs) require more intelligent analysis methods.

[0054] The present application integrates multi-source data through the agent based on the Dify platform, realizes efficient network traffic analysis and internal network terrain analysis, and improves network security capabilities.

[0055] It can be seen that the disadvantages of the prior art include:

[0056] High false positive rate: rule-based detection is prone to false positives, increasing analysis costs.

[0057] Insufficient real-time performance: static terrain analysis cannot reflect network changes.

[0058] Data silos: traffic data and topology data are separated, limiting comprehensive analysis.

[0059] Complex deployment: AI solutions require a lot of customization, with high implementation thresholds.

[0060] The present application integrates traffic analysis and terrain analysis through the agent of the Dify platform, providing a low-threshold, high-real-time network security solution, aiming to reduce false positives, improve threat response speed, and enhance network visibility.

[0061] The application provides a network traffic analysis and intranet terrain analysis method based on a Dify agent. The Dify platform is an open-source AI platform that supports the rapid construction of agents through workflow orchestration. The scheme utilizes the workflow node, tool integration, and model calling functions of Dify to deploy an agent, which includes the following modules: data collection, data processing, analysis, automated response, and training optimization. The overall process is shown in Figure 1 The method can include the following steps:

[0062] S1, collecting real-time and historical data from network devices, logs, and configurations; wherein the data includes network traffic logs, system logs, configuration files, and real-time network data;

[0063] S2, performing cleaning, standardization, and feature extraction operations on the collected data; wherein it specifically includes removing duplicate records, filling missing values, converting logs of different formats into a unified structure, and extracting traffic rate, connection frequency, device role, and log keyword key features;

[0064] S3, performing traffic analysis, terrain analysis, and anomaly detection on the processed data to identify abnormal behavior and potential threats in the network; wherein the Isolation Forest algorithm is used to detect abnormal traffic, the XGBoost algorithm is used to classify known threats, the GNN is used to construct network topology, and the K-Means clustering and ARIMA model are used to identify unknown anomalies and traffic mutations;

[0065] In the embodiments of the application, after performing traffic analysis, terrain analysis, and anomaly detection on the processed data, the method further includes:

[0066] training the agent model and optimizing the model parameters online to adapt to new threats and changes in the network environment, the training and optimization including training the Isolation Forest and XGBoost models using public data sets, optimizing the parameters of the GNN, and updating the model through incremental learning, the feedback loop node of Dify records false positives / false negatives, and adjusts the model weights.

[0067] S4, triggering response actions according to the analysis and detection results; wherein the response actions at least include isolating abnormal devices, blocking malicious IPs, and notifying administrators;

[0068] S5, generating a risk score and a topology graph according to the analysis and detection results; wherein it includes assigning grades to the generated statistics and calculating the risk score by weighting, and generating a network topology graph.

[0069] The embodiments of the application provide a network traffic and intranet terrain analysis system based on a Dify agent. The system can include:

[0070] Data collection module, for collecting real-time and historical data from network devices, logs and configurations; wherein the data includes network traffic logs, system logs, configuration files and real-time network data;

[0071] Data processing module, for cleaning, standardizing and feature extraction operations on the collected data; wherein it specifically includes removing duplicate records, filling missing values, converting different formats of logs into a unified structure, and extracting traffic rate, connection frequency, device role and log keyword key features;

[0072] Analysis module, for performing traffic analysis, terrain analysis and anomaly detection on the processed data to identify abnormal behavior and potential threats in the network; wherein the Isolation Forest algorithm is used to detect abnormal traffic, the XGBoost algorithm is used to classify known threats, the GNN is used to construct network topology, and the K-Means clustering and ARIMA model are used to identify unknown anomalies and traffic mutations;

[0073] Training and optimization module, for training agent models and optimizing model parameters online to adapt to new threats and changes in network environment, including training Isolation Forest and XGBoost models using public data sets, optimizing GNN parameters, and updating models through incremental learning, Dify feedback loop node records false positives / false negatives, and adjusts model weights.

[0074] Automatic response module, for triggering response actions according to analysis and detection results. Wherein the response actions at least include isolating abnormal devices, blocking malicious IP and notifying administrators;

[0075] Output result module, for generating risk score and topology graph according to analysis and detection results. Wherein it includes assigning grades to generated statistics and calculating risk score by weighting, and generating network topology graph.

[0076] The following gives the specific implementation process of the embodiments of the present application:

[0077] I. Data collection module

[0078] Function: Collect real-time and historical data from multiple network sources.

[0079] Data types (see Table 1):

[0080] Network traffic logs: such as NetFlow records, PCAP packets.

[0081] System logs: authentication logs, access logs.

[0082] Configuration files: router, switch configurations.

[0083] Real-time network data: SNMP data, ARP table.

[0084] Implementation: Configure data collection nodes in Dify workflow, obtain data through APIs (e.g., Zeek, Wireshark) or log collection tools (e.g., Filebeat).

[0085] Generated statistics: Data collection frequency (times / minute), data source coverage (collected data types / total types).

[0086] Table 1 Network data type table

[0087]

[0088] II. Data processing module

[0089] Function: Cleaning, standardization, and feature extraction, preparing data for analysis module.

[0090] Processing logic:

[0091] Cleaning: Remove duplicate records, fill in missing values (e.g., logs with empty IP addresses, use context inference).

[0092] Standardization: Convert different formats of logs (e.g., JSON, CSV) to a unified structure (e.g., Parquet).

[0093] Feature extraction: Extract traffic rate, connection frequency, device role (server / client), log keywords (e.g., "login failure").

[0094] Dify implementation: Configure data processing nodes, use Python tools (e.g., Pandas) for cleaning, NLP nodes (call Dify integrated LLM, such as LLaMA) to parse log text and extract event keywords.

[0095] Generated statistics: Traffic rate (bytes / second), connection frequency (times / minute), log event count, device activity (active time / total time).

[0096] III. Analysis module

[0097] Function: Perform traffic analysis, terrain analysis, and anomaly detection.

[0098] Sub-modules include:

[0099] (1) Traffic analysis:

[0100] Algorithm: Isolation Forest (unsupervised learning) detects abnormal traffic, XGBoost (supervised learning) classifies known threats (e.g., DDoS, malicious scanning).

[0101] Features: traffic size, port distribution, protocol type, time series pattern.

[0102] Statistical Items: abnormal traffic score (0-1, based on Isolation Forest anomaly probability), threat type label (e.g., "malicious", "normal").

[0103] (2) Intranet Terrain Analysis:

[0104] Algorithm: Graph Neural Network (GNN) constructs network topology, analyzes node (device) and edge (connection) relationships.

[0105] Features: device IP, MAC address, connection relationship, VLAN information.

[0106] Statistical Items: topology change frequency (number of changes per hour), key node importance (based on PageRank algorithm).

[0107] (3) Anomaly Detection:

[0108] Algorithm: K-Means clustering identifies unknown anomalies, ARIMA model detects traffic mutations.

[0109] Features: traffic rate changes, log event frequency.

[0110] Statistical Items: abnormal event count, risk score (combines traffic and log analysis).

[0111] Dify Implementation: Configure analysis nodes, call open-source models (such as IsolationForest from Scikit-learn, GNN from PyTorch) or Dify-integrated LLM for semantic analysis (such as parsing attack intent in logs).

[0112] Process: As shown in Figure 2 , the analysis module receives processed data, performs parallel analysis, and outputs statistical items to the scoring module.

[0113] Key Items:

[0114] Application Log Data:

[0115] Core Fields: device ID, application name, event type, timestamp, user ID, request URL.

[0116] Data Description: Application logs reflect software behavior on devices, such as access records of WEB servers, database query logs, or event logs of security software. These data can reveal potential abnormal behavior, such as frequent error logs that may indicate attack attempts.

[0117] Processing Logic: Configure log collection nodes through Dify workflow, use tools like Logstash to extract logs from applications like Apache, MySQL. Tokenize log text, call Dify's NLP node (based on LLaMA) to extract keywords like "error" "access denied", count event frequency and abnormal patterns.

[0118] Generated Statistics: Application event frequency (times / hour), abnormal event count (like error log number), keyword matching rate (abnormal keywords / total keywords).

[0119] Device Fingerprint Data:

[0120] Core Fields: Device ID, MAC address, operating system version, installed software list, hardware serial number, timestamp.

[0121] Data Description: Device fingerprint data records static and dynamic characteristics of devices, which can be used to detect device disguising or abnormal configuration (such as unauthorized software installation).

[0122] Processing Logic: Configure fingerprint collection nodes through Dify workflow, use tools like Nmap, Osquery to scan device features. Compare device fingerprints with historical records to detect changes (such as operating system version changes). Match the installed software list to identify high-risk software (such as known malicious software).

[0123] Generated Statistics: Fingerprint change frequency (number of changes per day), high-risk software installation times, device consistency score (similarity between current and historical fingerprints).

[0124] Traffic Analysis Details:

[0125] Isolation Forest detects unknown anomalies, generates scores based on anomaly probability; XGBoost classifies known threats (such as DDoS, SQL injection) using pre-trained models.

[0126] Feature Selection: Traffic size (bytes / second), port distribution (proportion of commonly used ports), protocol type (TCP / UDP ratio), time series pattern (traffic peak).

[0127] Processing Logic: Dify analysis node calls Isolation Forest of Scikit-learn, inputs traffic features, outputs anomaly score (0-1). XGBoost model is trained using CICIDS2017 dataset, inputs the same features, outputs threat label (such as "malicious" "normal"). When the anomaly score is higher than 0.8 or the threat label is "malicious", trigger high-risk alert.

[0128] Generated statistics: Anomalous traffic score, threat type label, traffic peak frequency (peaks / hour).

[0129] Terrain analysis details:

[0130] Algorithm: Graph Neural Network (GNN) analyzes device and connection relationships, PageRank algorithm calculates node importance.

[0131] Feature selection: Device IP, MAC address, connection relationships (source-destination IP pairs), VLAN information, device roles (server / client).

[0132] Processing logic: Dify analysis node invokes GNN model from PyTorch, inputs device and connection data, generates topology graph. PageRank algorithm identifies key nodes (e.g., servers, importance score > 0.5). Detects topology changes (e.g., new device joins), records change frequency.

[0133] Generated statistics: Topology change frequency, key node importance, connection density (number of edges / node).

[0134] Anomaly detection details:

[0135] Algorithm: K-Means clustering identifies unknown anomalies, ARIMA model detects traffic time series mutations.

[0136] Feature selection: Traffic rate changes (standard deviation), log event frequency (times / hour), topology change frequency, device consistency score.

[0137] Processing logic: Dify analysis node invokes K-Means from Scikit-learn, inputs comprehensive features, identifies anomaly clusters (distance from centroid > threshold). ARIMA model analyzes traffic rate series, detects mutations (e.g., spikes > 2x mean). Integrates traffic analysis and terrain analysis statistics to generate risk score.

[0138] Generated statistics: Anomalous event count, risk score, mutation frequency (mutations / hour).

[0139] Four, Automated Response Module

[0140] Function: Trigger response actions based on analysis results.

[0141] Actions:

[0142] Isolate anomalous devices (invoke firewall API, e.g., Palo Alto).

[0143] Block malicious IPs (update access control list, ACL).

[0144] Notify administrator (sent to Slack via Dify's Webhook node).

[0145] Dify implementation: Configure automation nodes to integrate external tools (e.g., firewall, SIEM system).

[0146] Five, training and optimization module

[0147] Function: Initial training and online optimization of agent model.

[0148] Implementation:

[0149] Initial training: Train Isolation Forest and XGBoost models using public datasets (e.g., CICIDS2017) and optimize GNN parameters.

[0150] Online optimization: Update models through incremental learning, Dify's feedback loop node records false positives / misreports, and adjusts model weights.

[0151] Statistical items: Model accuracy (correct predictions / total predictions), false positive rate (false positives / total alerts).

[0152] Six, scoring and weighting

[0153] The analysis module generates statistical items, assigns grades (high / medium / low) and calculates risk scores (see Table 2) by weighting. The weights are assigned according to the impact of statistical items on security, for example, the abnormal traffic score weight is higher (6%).

[0154] Table 2: Statistical item grade and weight table

[0155]

[0156] Calculation process: Convert statistical item grades to numerical values (high = 3, medium = 2, low = 1), and sum the weighted values to get the risk score. For example, if the abnormal traffic score is high (3) and the topology change is low (1), the risk score = 3x6% + 1x3% +….

[0157] In summary, the key points and points to be protected of the present application include:

[0158] Dify agent integration: Use Dify workflow to achieve seamless integration of traffic analysis and terrain analysis.

[0159] Multi-algorithm collaboration: Combine Isolation Forest, XGBoost and GNN to enhance analysis accuracy.

[0160] Real-time dynamics: Agent processes data in real time and updates topology map.

[0161] Multi-source data fusion: Integrates traffic, log, configuration and real-time data to provide comprehensive insights.

[0162] Adaptive model optimization: Online learning through Dify's feedback loop node to dynamically adjust model parameters, adapting to new threats and network environment changes.

[0163] Scalability and ease of use: Dify platform supports modular expansion, allowing users to customize workflows based on network size, suitable for small and medium-sized enterprises to large enterprises.

[0164] In an optional embodiment of the present application, traditional tools include the use of Snort (IDS) and Nmap (scanning), but have poor integration and real-time performance. Other AI platforms include TensorFlow or PyTorch, which require manual workflow setup and lack the convenience of Dify.

[0165] In one embodiment, an electronic device, which can be a computer, has an internal structure diagram as shown in Figure 3 The electronic device includes a processor, a memory, and a network interface connected by a system bus. The processor of the device provides computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for network traffic and internal terrain analysis data based on Dify agent. The network interface of the computer device is used for communication with external terminals through network connection. The computer program is executed by the processor to implement a network traffic and internal terrain analysis method based on Dify agent.

[0166] Those skilled in the art can understand that the structure shown in Figure 3 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] In one embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by the processor to implement the steps of the above-mentioned network traffic and internal terrain analysis method based on Dify agent.

[0168] In one embodiment of the present application, a computer program product is provided, which includes computer programs / instructions. The computer program is executed by the processor to implement the steps of the above-mentioned network traffic and internal terrain analysis method based on Dify agent.

[0169] The computer-readable storage medium and computer program product provided in this embodiment are similar in implementation principle and technical effect to the above method embodiments, and will not be repeated here.

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for network traffic and intranet terrain analysis based on Dify agents, characterized in that, The method includes: Real-time and historical data are collected from network devices, logs, and configurations; wherein the data includes network traffic logs, system logs, configuration files, and real-time network data; The collected data is cleaned, standardized, and feature extracted. Specifically, this includes removing duplicate records, filling in missing values, converting logs of different formats into a unified structure, and extracting key features such as traffic rate, connection frequency, device role, and log keywords. The processed data is subjected to traffic parsing, terrain analysis, and anomaly detection to identify anomalous behavior and potential threats in the network. Specifically, the Isolation Forest algorithm is used to detect anomalous traffic, the XGBoost algorithm is used to classify known threats, GNN is used to construct the network topology, and K-Means clustering and ARIMA model are used to identify unknown anomalies and traffic mutations. The response action is triggered based on the analysis and detection results; the response action includes at least isolating abnormal devices, blocking malicious IPs, and notifying the administrator; Based on the analysis and detection results, risk scores and topology diagrams are generated; this includes assigning levels to the generated statistical items and calculating risk scores with weights, as well as generating network topology diagrams.

2. The method according to claim 1, characterized in that, After performing traffic parsing, terrain analysis, and anomaly detection on the processed data, the method further includes: The agent model is trained and its parameters are optimized online to adapt to new threats and changes in the network environment. The training and optimization include training Isolation Forest and XGBoost models using public datasets, optimizing the parameters of the GNN, updating the model through incremental learning, recording false positives / false negatives in the feedback loop node of Dify, and adjusting the model weights.

3. The method according to claim 1, characterized in that, Real-time and historical data are collected from network devices, logs, and configurations, including: Real-time and historical data are collected by configuring data acquisition nodes in the Dify workflow and obtaining data through APIs or log collection tools; the data acquisition further includes configuring data acquisition nodes to obtain data from multiple network sources, performing data acquisition frequency statistics, and performing data source coverage statistics.

4. The method according to claim 1, characterized in that, Data cleaning, standardization, and feature extraction are performed using Dify, specifically including: Use Python tools for data cleaning, including removing duplicate records and filling in missing values; Use NLP nodes to parse log text and extract event keywords; Convert logs of different formats into a unified structure; Extract traffic rate, connection frequency, device role, and log keyword features; Generate statistics, including traffic rate, connection frequency, log event count, and device activity.

5. The method according to claim 1, characterized in that, The processed data undergoes traffic parsing, terrain analysis, and anomaly detection, including: Anomaly traffic is detected using the Isolation Forest algorithm, and scores are generated based on the anomaly probability. The XGBoost model uses a pre-trained model, inputting traffic features and outputting threat labels. The GNN model analyzes device and connection relationships, while the PageRank algorithm calculates node importance. K-Means clustering is used to identify unknown anomalies, and the ARIMA model is used to detect sudden changes in traffic time series. The system receives and processes the data, performs parallel analysis, and outputs statistical items, including abnormal traffic scores, threat type labels, topology change frequency, and critical node importance.

6. The method according to claim 1, characterized in that, The response actions triggered based on the analysis and detection results include: Use the firewall API to isolate abnormal devices and update the access control list to block malicious IPs; Send notifications to the administrator via Dify's Webhook node; Configure automation nodes and integrate external tools to achieve automated responses; The response action is triggered based on the analysis results.

7. A network traffic and intranet terrain analysis system based on Dify agents, characterized in that, The system includes: The data acquisition module is used to collect real-time and historical data from network devices, logs, and configurations; wherein, the data includes network traffic logs, system logs, configuration files, and real-time network data; The data processing module is used to clean, standardize, and extract features from the collected data. Specifically, this includes removing duplicate records, filling in missing values, converting logs of different formats into a unified structure, and extracting key features such as traffic rate, connection frequency, device role, and log keywords. The analysis module performs traffic parsing, topography analysis, and anomaly detection on the processed data to identify anomalous behavior and potential threats in the network. Specifically, it uses the Isolation Forest algorithm to detect anomalous traffic, the XGBoost algorithm to classify known threats, GNN to construct network topology, and K-Means clustering and ARIMA model to identify unknown anomalies and traffic mutations. An automated response module is used to trigger response actions based on the analysis and detection results; wherein, the response actions include at least isolating abnormal devices, blocking malicious IPs, and notifying the administrator; The output results module is used to generate risk scores and topology maps based on the analysis and detection results; this includes assigning levels to the generated statistical items and calculating risk scores with weights, as well as generating network topology maps.

8. The system according to claim 7, characterized in that, The system also includes: The training and optimization module is used to train the agent model and optimize the model parameters online to adapt to new threats and changes in the network environment. The training and optimization include training Isolation Forest and XGBoost models using public datasets, optimizing the parameters of GNN, updating the model through incremental learning, and Dify's feedback loop node recording false positives / false negatives and adjusting model weights.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images