Full-flow analysis method and device

Abstract

The invention relates to the technical field of flow analysis, in particular to a full flow analysis method and device. The method comprises the following steps: capturing full traffic in a network environment based on a DPDK framework, and processing the full traffic to obtain the processed full traffic; analyzing the processed full flow to obtain at least one original data packet; establishing a mapping relationship between each original data packet and the first index information corresponding to the original data packet; performing real-time analysis on each original data packet to obtain a real-time analysis result corresponding to each original data packet; retrieving a corresponding target data packet from the at least one original data packet based on the analysis request and the mapping relationship; performing cross-time-dimension behavior analysis on the target data packet to obtain a deep analysis result; and integrating the real-time analysis result and the deep analysis result corresponding to the target data packet to generate a network state analysis report. According to the method, the full flow can be captured, and the full flow can be stored and analyzed to meet user requirements.

Description

Technical Field

[0001] This invention relates to the field of flow analysis technology, and in particular to a method and apparatus for full flow analysis. Background Technology

[0002] In today's enterprise network and internet environments, network failures, security attacks, and performance bottlenecks occur frequently. To diagnose these problems, operations and maintenance personnel typically need to capture and analyze network traffic to pinpoint the cause of the anomaly. Currently, mainstream traffic capture and analysis tools, such as Wireshark, tcpdump, and Tshark, can capture and analyze data packets on specific network interface cards (NICs). They offer various functions, including protocol parsing, filtering, and statistics. However, in practical applications, these tools face some challenges.

[0003] First, traditional packet capture tools have limited capabilities in traffic capture, making it difficult to achieve full traffic capture. Most traditional packet capture tools are developed based on the libpcap (Packet Capture Library) library. These tools interact with the operating system kernel through libpcap, utilizing the packet capture mechanisms provided by the kernel's network protocol stack to obtain network traffic. This multi-layered architecture, relying on user-space libraries (such as libpcap), the kernel protocol stack, and specific capture interfaces, introduces limitations in high-traffic environments (e.g., above 10Gbps). For example, with limited buffer size, packet loss may occur, making it impossible to completely record all passing packets. Furthermore, traditional packet capture tools typically only capture traffic through specific ports, failing to cover all network traffic, especially traffic across VLANs or multiple network interface cards (NICs). Moreover, traditional packet capture tools are primarily used for real-time analysis; the captured data is often temporarily stored, which is not conducive to long-term preservation and retrospective analysis.

[0004] Furthermore, while traditional packet capture tools can provide real-time alerts based on rule matching and support packet parsing, they either lack evidence at the raw packet level or cannot effectively correlate the parsed packet results with real-time alerts. This means that operations and maintenance personnel may need to manually compare time and logs to complete the analysis, increasing the complexity and difficulty of the task.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention proposes a full traffic analysis method and apparatus, which can capture, store and analyze the full traffic to meet user needs.

[0007] The embodiments of the present invention adopt the following technical solutions: In a first aspect, the present invention provides a full traffic analysis method, specifically comprising: capturing full traffic data in a network environment based on the DPDK framework; processing the full traffic data to obtain processed full traffic data; and parsing the processed full traffic data to obtain at least one original data packet. Establish a mapping relationship between each of the original data packets and its corresponding first index information; Each of the original data packets is analyzed in real time to obtain the real-time analysis results corresponding to each of the original data packets; Based on the analysis request and the mapping relationship, the corresponding target data packet is retrieved from at least one original data packet; cross-time dimension behavior analysis is performed on the target data packet to obtain in-depth analysis results; By integrating the real-time analysis results and the in-depth analysis results corresponding to the target data packets, a network status analysis report is generated.

[0008] Preferably, establishing a mapping relationship between each of the original data packets and its corresponding first index information; and storing the original data packets and the first index information respectively includes: All raw data packets are grouped into fixed time windows, and a base timestamp for each window is calculated. For each group, all the original data packets contained in the group are written consecutively into a packet file; the filename of the packet file includes the base timestamp. Create a corresponding first index entry for each raw data packet; For each group, the first index entry corresponding to the original data packet contained in the group is written into an original index file, and the file name of the original index file contains the basic timestamp. The base timestamp is converted to local time, and the local time and the storage path of the corresponding original index file are stored in the database as the first index information.

[0009] Preferably, the first index entry includes a first index value, and establishing a corresponding first index value for each original data packet includes: Extract the feature information from the original data packet; The feature information is converted into multiple corresponding decimal values, and the decimal values ​​are used as the first index value of the original data packet.

[0010] Preferably, the first index entry further includes the offset and length of the original data packet in the corresponding packet file, and the step of retrieving the corresponding target data packet from at least one original data packet based on the analysis request and the mapping relationship includes: Parse the time range conditions and protocol feature conditions in the analysis request; Obtain second index information that meets the time range conditions; Locate the corresponding target index file in the file system based on the second index information; The second index entry in the target index file is filtered to obtain the offset and length that match the protocol feature conditions; Based on the offset and length, the corresponding original data packet is read from the packet file, and the original data packet is encapsulated into a standard pcap format to obtain the target data packet.

[0011] Preferably, the deep analysis results include session records and field information, and the step of performing cross-time dimension behavioral analysis on the target data packet to obtain the deep analysis results includes: The target data packet is subjected to protocol layer parsing to identify the protocol information of the link layer, network layer, and transport layer; Based on the protocol information, the sessions in the target data packet are classified and statistically analyzed to generate session records for dialogues at each level; Each frame of data in the target data packet is parsed to obtain field information containing the content of the target data packet.

[0012] Preferably, the real-time analysis results include traffic classification results, and the step of performing real-time analysis on each of the original data packets to obtain the real-time analysis results corresponding to each of the original data packets includes: Each of the original data packets is decoded layer by layer to obtain the protocol type and protocol information used by each original data packet at different layers; For each raw data packet, the protocol type and protocol information corresponding to each layer are combined to obtain the protocol stack combination corresponding to the raw data packet. Based on the protocol stack combination, the traffic of the original data packet is classified to obtain the traffic classification result.

[0013] Preferably, the real-time analysis results further include application evaluation results, and the parallel real-time analysis of the original data packets to obtain the real-time analysis results includes: Extract application layer data related to the target application service from the original data packet; Identify the protocol type used by the application layer data; Based on the protocol type, a corresponding protocol parser is selected, and behavioral analysis is performed on the application layer data to obtain key performance indicators that reflect the operational status of the target application business. The health status of the target application service is assessed based on the key performance indicators, and the application assessment results are obtained.

[0014] Preferably, the step of integrating the real-time analysis results and the deep analysis results corresponding to the target data packet to generate a network status analysis report includes: Obtain the real-time analysis results corresponding to the target data packet, wherein the real-time analysis results include the target timestamp of the abnormal event; Obtain the deep analysis results corresponding to the target timestamp, the deep analysis results including the protocol violation field; Based on the target timestamp, abnormal events and protocol violation fields are integrated to generate a network status analysis report.

[0015] In a second aspect, the present invention provides a full flow analysis apparatus, specifically comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to perform the full flow analysis method of the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect.

[0017] Compared with existing technologies, the advantages of this invention are as follows: It achieves full-traffic packet loss-free capture through the DPDK framework, effectively avoiding the performance bottlenecks of traditional kernel protocol stacks. User-space polling and memory pool optimization techniques ensure zero packet loss even at 100Gbps traffic levels. Simultaneously, it constructs a time-series index storage architecture, precisely associating raw data packets with index information in the time dimension. This allows users to quickly locate and extract the corresponding data packet content based on time range, protocol characteristics, and other conditions, achieving millisecond-level response and access. Furthermore, it integrates corresponding real-time analysis results and deep analysis results based on user needs, achieving efficient collaboration between real-time and deep analysis while ensuring full-traffic storage. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is a flowchart illustrating a full flow analysis method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating step 102 provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a process for retrieving a corresponding target data packet from at least one original data packet based on an analysis request and the mapping relationship provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of a process for performing cross-time dimension behavior analysis on a target data packet, provided by an embodiment of the present invention; Figure 5 This is a schematic diagram of a process for real-time analysis of raw data packets provided by an embodiment of the present invention; Figure 6 This is a schematic diagram of another process for real-time analysis of raw data packets provided by an embodiment of the present invention; Figure 7 This is a flowchart for generating alarms provided in an embodiment of the present invention; Figure 8 This is a flowchart of baseline generation provided in an embodiment of the present invention; Figure 9 This is a flowchart of report generation provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a full flow analysis device provided in an embodiment of the present invention; The reference numerals in the attached figures are as follows: 21: Processor; 22: Memory. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0022] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0023] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0024] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0025] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Example 1: This invention provides a full traffic analysis method that is independently implemented by multiple modules, reducing coupling and making it easier to troubleshoot and expand. These modules include a traffic capture module, a real-time analysis module, a packet storage module, a data statistics module, a deep analysis module, and a data display module. By separating packet capture, parsing, and storage into independent processes with shared memory, packet capture, parsing, and storage can be performed in real-time and simultaneously. For packet storage, high-speed packet writing is achieved using Linux's AIO (Asynchronous Input / Output) method, combined with system file characteristics to ensure real-time packet storage.

[0027] like Figure 1 As shown, Embodiment 1 of the present invention provides a full flow analysis method, which specifically includes the following steps: Step 101: Capture full traffic data in the network environment based on the DPDK framework, process the full traffic data to obtain processed full traffic data; parse the processed full traffic data to obtain at least one raw data packet.

[0028] In one embodiment, DPDK (Data Plane Development Kit) is a high-performance data plane development toolset. DPDK directly manages the data transmission and reception of the network interface card (NIC) through user-space drivers, cleverly bypassing the performance bottlenecks of traditional kernel protocol stacks to achieve efficient full-traffic capture. DPDK utilizes Userspace I / O (UIO) technology to map the NIC device to user space, reducing the switching latency between kernel and user modes. It also employs a polling mechanism to proactively acquire full-traffic data, further reducing processing latency and increasing throughput. Furthermore, DPDK optimizes memory access speed by using large page memory to reduce Translation Lookaside Buffer Misses (TLBMiss) and eliminates performance fluctuations caused by dynamic memory allocation through pre-allocated memory pools and zero-copy technology. To maximize performance, DPDK also ensures that the NIC, CPU core, and memory reside on the same node, reducing latency associated with cross-node access. These design features enable DPDK to perform exceptionally well even in high-traffic environments. Real-world testing shows that it achieves near-zero packet loss in 100Gbps networks, providing a reliable foundation for subsequent data storage and analysis. Furthermore, the entire traffic data is processed, including filtering irrelevant packets, removing duplicates, and segmenting traffic types. The processed packets are then pushed in a specific format to the real-time analysis module and the packet storage module for subsequent storage and analysis. The real-time analysis module and the packet storage module are independent processes.

[0029] In one embodiment, multiple queues can be configured according to the network interface card (NIC) queue balancing strategy, binding each receive queue to a different CPU core. This fully utilizes the capabilities of multi-core processors, reduces the load on individual cores, and thus improves the overall input / output (I / O) performance of DPDK. By parsing the source Internet Protocol (IP), destination IP, source port, destination port, and protocol type of the raw data packet and forming a five-tuple, the data packet is distributed to different queues according to the strategy. This ensures that the data packets are evenly distributed across the queues, avoiding overload of any particular queue, and allows multiple cores to process the data packets in these queues in parallel, further improving the processing speed of DPDK. The five-tuple includes the source IP, destination IP, source port, destination port, and protocol type.

[0030] Step 102: Establish a mapping relationship between each of the original data packets and its corresponding first index information; and store the original data packets and the first index information respectively.

[0031] In one embodiment, by establishing a relationship between the original data packet, index entry, and index information, the original data packet and index information are linked together, and the three are linked by a timestamp. This enables fast retrieval and analysis while ensuring storage efficiency. This reduces storage redundancy and supports complex queries at the millisecond level, such as Hypertext Transfer Protocol (HTTP) requests from a certain IP address within a specific time period, thereby resolving the contradiction between massive data storage and efficient analysis in full-traffic scenarios.

[0032] Step 103: Perform real-time analysis on each of the original data packets to obtain the real-time analysis results corresponding to each of the original data packets.

[0033] In one embodiment, the present invention employs a parallel processing architecture to perform real-time analysis on all raw data packets. The real-time analysis includes basic analysis and application analysis. The real-time analysis results include traffic classification results and application evaluation results. Basic analysis (parsing the network protocol stack of the link layer, network layer, and transport layer, and classifying the traffic) and application analysis (calling parsers of different protocols to parse application layer protocols and obtain business health status) can run simultaneously and independently. The interruption of any analysis task will not affect the continuous execution of the other task. All real-time analysis results are written to the database in real time, providing complete data support for subsequent correlation queries.

[0034] Step 104: Based on the analysis request and the mapping relationship, retrieve the corresponding target data packet from at least one original data packet; perform cross-time dimension behavior analysis on the target data packet to obtain in-depth analysis results.

[0035] In one embodiment, deep analysis employs an on-demand triggering mechanism. When a user submits an analysis request containing conditions such as time range and protocol characteristics, the system quickly retrieves target data packets that meet the conditions from the file system based on the request and performs cross-time dimension protocol behavior analysis to obtain deep analysis results. These deep analysis results are also stored in a database.

[0036] Step 105: Integrate the real-time analysis results and the in-depth analysis results corresponding to the target data packet to generate a network status analysis report.

[0037] In one embodiment, the target data packet is the original data packet that meets the analysis request. In step 103, the real-time analysis results of all the original data packets have been stored in the database. The real-time analysis results corresponding to the target data packet are obtained from the database. The real-time analysis results include the target timestamp of the abnormal event. The deep analysis results corresponding to the target timestamp are obtained from the database. The deep analysis results include the protocol violation field. The abnormal event and the protocol violation field are integrated based on the target timestamp to generate a network status analysis report.

[0038] This embodiment achieves full-traffic packet loss-free capture through the DPDK framework, effectively avoiding the performance bottlenecks of traditional kernel protocol stacks. It employs user-space polling and memory pool optimization techniques to ensure zero packet loss even at 100Gbps traffic levels. Simultaneously, it constructs a time-series index storage architecture, precisely associating raw data packets with index information along the time dimension. This allows users to quickly locate and extract corresponding data packet content based on time range, protocol characteristics, and other conditions, achieving millisecond-level response and access. Furthermore, it integrates corresponding real-time and deep analysis results based on user needs, achieving efficient collaboration between real-time and deep analysis while ensuring full-traffic storage.

[0039] In high-traffic scenarios, to achieve in-depth analysis of all network traffic data, it is first necessary to capture and store these raw data packets. When specific analysis is required, the target data packets are extracted from the file system and then parsed and processed. However, due to the extremely large number of raw data packets, directly retrieving these massive amounts of raw packets based on content or metadata would result in very slow response times, making it difficult to meet the needs of real-time or near real-time analysis. This embodiment achieves efficient storage and sub-second retrieval of raw data packets through a time window and hierarchical indexing method.

[0040] like Figure 2 As shown, this embodiment provides a specific implementation method for step 102, which specifically includes the following steps: Step 201: Group all raw data packets into fixed time windows and calculate the base timestamp for each window.

[0041] In one embodiment, a fixed time window refers to a pre-set, continuous time interval of fixed length (such as 1 minute, 5 minutes, or 1 hour), and the base timestamp of each window refers to the start time of this fixed time window. The capture timestamp of each raw data packet is obtained, and all raw data packets whose capture timestamps fall within the same window interval are grouped together. The fixed time window can be freely set according to the actual situation and is not limited here.

[0042] Step 202: For each group, write all the original data packets contained in the group into a packet file consecutively; the file name of the packet file includes the base timestamp.

[0043] In one embodiment, the raw data packets of the same group are written into a packet file in the order of their capture timestamps, with one packet file corresponding to one group. All packet files are stored in the file system, and the filename of each packet file contains the base timestamp. For example, a packet file can be named "1620000000.-d".

[0044] Step 203: Create a corresponding first index entry for each raw data packet.

[0045] In one embodiment, the first index entry includes a first index value and the offset and length of the original data packet in the corresponding packet file. The method for obtaining the first index value is as follows: extracting feature information from the original data packet; converting the feature information into multiple corresponding decimal values, and using the decimal values ​​as the first index value of the original data packet.

[0046] Step 204: For each group, write the first index entry corresponding to the original data packet contained in the group into an original index file, wherein the file name of the original index file contains the basic timestamp.

[0047] In one embodiment, one group corresponds to one raw index file. All raw index files are written to the file system. The filename of each raw index file contains a base timestamp. For example, a raw index file can be named "1620000000.-i". The raw index files are associated with the package files through the base timestamp.

[0048] Step 205: Convert the base timestamp to local time, and store the local time and the storage path of the corresponding original index file as the first index information in the database.

[0049] In one embodiment, the storage path of the original index file corresponding to the base timestamp is obtained by mapping the file name of the original index file to the base timestamp, and the base timestamp is converted to local time according to the conversion rules.

[0050] Furthermore, the package file, the original index file, and the first index information are all written using an asynchronous input / output interface, and this write operation is performed by an independent process to ensure stability.

[0051] Meanwhile, this invention provides a dedicated reading service (or library function) that supports rapid retrieval of stored raw data packets based on multi-dimensional conditions such as time range and protocol characteristics, thereby achieving an efficient read-write separation full-traffic data management architecture.

[0052] like Figure 3 As shown, this embodiment provides a method for retrieving a corresponding target data packet from at least one original data packet based on an analysis request and the mapping relationship. The method specifically includes the following steps: Step 301: Parse the time range conditions and protocol feature conditions in the analysis request.

[0053] In one embodiment, protocol characteristics include one or more of the following: source IP, destination IP, source port, destination port, identifier, or protocol.

[0054] Step 302: Obtain second index information that meets the time range conditions from the database.

[0055] In one embodiment, in the database, using time as the search condition, second index information that meets the time range condition is retrieved from multiple first index information. For example, if the time range is from 2021-05-04 00:00:00 to 2021-05-04 00:00:59.999, and 1620000000 converts to local time 2021-05-04 00:00:00, which falls within the time range, then the corresponding second index information can be found.

[0056] Step 303: Locate the corresponding target index file in the file system based on the second index information.

[0057] In one embodiment, based on the retrieved second index information, the storage path of the second index information record is obtained, and the corresponding target index file is located in the file system according to the path. That is, the target index file is the original index file that meets the time range condition.

[0058] Step 304: Filter the second index entries in the target index file to obtain the offset and length that match the protocol feature conditions.

[0059] In one embodiment, the second index entries in the target index file found in step 303 are matched item by item to find index entries that meet the protocol conditions, and the offset and length of the original data packet corresponding to the index entry in the corresponding packet file are obtained.

[0060] Step 305: Based on the offset and length, read the corresponding original data packet from the packet file, encapsulate the original data packet into a standard pcap format, and obtain the target data packet.

[0061] In one embodiment, the corresponding packet file is matched based on the base timestamp on the filename of the target index file. The corresponding raw data packet is read from the packet file based on the offset and length, and then encapsulated into standard pcap format to obtain the target data packet. If the raw data packet is in another format, it needs to be converted to pcap format.

[0062] After obtaining the target data packet, it is possible to conduct in-depth analysis based on the user's network security needs, detect whether there are protocol violation fields, such as illegal flag bits or abnormal field values; at the same time, combined with session classification statistics, the communication patterns and data interaction characteristics can be analyzed to reconstruct the causal chain of network events and identify potential security threats.

[0063] like Figure 4 As shown, this embodiment provides a method for performing cross-time dimension behavior analysis on target data packets. The method specifically includes the following steps: Step 401: Perform protocol layer parsing on the target data packet to identify the protocol information of the link layer, network layer, and transport layer.

[0064] In one embodiment, when performing protocol layered parsing on a target data packet, a bottom-up approach can be used to peel off the protocol header layer by layer. First, the MAC address (Media Access Control Address) of the link layer is identified. Then, key fields such as IP fragmentation and Time To Live (TTL) values ​​of the network layer are parsed. Finally, the port number, sequence number, and control flags of the transport layer are extracted. This layered decoding mechanism ensures the complete extraction of protocol features at each layer, providing accurate protocol context information for subsequent anomaly detection and traffic analysis.

[0065] Step 402: Based on the protocol information, classify and statistically analyze the sessions in the target data packet to generate session records for dialogues at each level.

[0066] In one embodiment, based on the extracted protocol information from each layer, the target data packet can be reassembled using a quintuple and time-series correlation algorithm. This automatically identifies and counts MAC address sessions at the link layer, IP communication pairs at the network layer, and TCP / UDP session flows at the transport layer, generating a complete session record table containing metrics such as session duration, number of data packets, and traffic size. At the same time, abnormal sessions (such as IP fragmentation attacks) are marked, providing structured session-level statistical data for network behavior analysis.

[0067] Step 403: Parse each frame of data in the target data packet to obtain field information containing the content of the target data packet.

[0068] In one embodiment, when performing frame-by-frame parsing of the target data packet, a protocol stack-aware decoding engine can be used to disassemble the data packet structure in the order of Ethernet frame header, IP packet header, and transport layer header, accurately extracting protocol fields at each layer (such as source MAC address, destination MAC address, IP fragmentation identifier, and TCP sequence number). At the same time, it supports in-depth parsing of application layer payloads (such as HTTP headers). This fine-grained field-level analysis can identify protocol violation fields that are difficult for traditional packet capture tools to detect.

[0069] To obtain richer information, real-time analysis can be performed while capturing full traffic. Real-time analysis can identify network anomalies immediately, providing a precise target range for subsequent in-depth analysis. This real-time preprocessing mechanism greatly improves the efficiency and timeliness of full traffic analysis, ensuring that important security events are not overwhelmed by massive amounts of data.

[0070] like Figure 5 As shown, this embodiment provides a method for real-time analysis of raw data packets, which specifically includes the following steps: Step 501: Decode the original data packet layer by layer to obtain the protocol type and protocol information used by the original data packet at different layers.

[0071] In one embodiment, the system uses protocol stack layered parsing technology to decode the original data packets layer by layer, starting with the MAC address and Ethernet type of the link layer, and then extracting key protocol information such as the IP version and TTL value of the network layer, and the port number and flag bits of the transport layer.

[0072] Step 502: Combine the protocol type and protocol information corresponding to each layer to obtain the protocol stack combination corresponding to the original data packet.

[0073] In one embodiment, by integrating the protocol types (such as Ethernet and TCP) and corresponding field information (such as MAC address, source IP and destination port) obtained from parsing the original data packets at each protocol layer, a complete protocol stack combination structure is constructed. This aggregation of protocol features from the link layer to the application layer not only restores the end-to-end communication path of the data packets, but also identifies complex communication patterns such as unconventional protocol nesting, providing an accurate protocol context basis for in-depth traffic analysis and anomaly detection.

[0074] Step 503: Classify the traffic of the original data packets based on the protocol stack combination to obtain the traffic classification result.

[0075] In one embodiment, based on the parsed complete protocol stack combination, the original data packets are automatically classified into specific service types and traffic categories by matching a predefined protocol feature library and a machine learning model. The traffic categories include web browsing, file transfer, email service, video streaming, voice communication, and database access.

[0076] like Figure 6 As shown, this embodiment provides another method for real-time analysis of raw data packets, which specifically includes the following steps: Step 601: Extract application layer data related to the target application service from the original data packet.

[0077] In one embodiment, the underlying protocol encapsulation in the original data packet is stripped to extract application layer data related to the target application business for subsequent application analysis. The application layer data includes HTTP requests, DNS (Domain Name System) queries, and database operations, etc.

[0078] Step 602: Identify the protocol type used by the application layer data.

[0079] In one embodiment, the specific protocol type used by the application layer data is identified based on protocol characteristics, port number, or the syntactic structure of the protocol content. Specific protocol types include HTTP, DNS, File Transfer Protocol (FTP), and Simple Mail Transfer Protocol (SMTP), etc.

[0080] Step 603: Select the corresponding protocol parser based on the protocol type, perform behavioral analysis on the application layer data, and obtain key performance indicators that reflect the operating status of the target application service.

[0081] In one embodiment, an appropriate protocol resolver is selected based on the protocol type used by the identified application layer data. Different protocol resolvers are specifically designed to parse the data structures and content of specific protocols. By performing in-depth analysis and behavioral analysis of application layer data using these protocol resolvers, key performance indicators (KPIs) reflecting the operational status of the target application can be extracted. Among these, KPIs related to the HTTP protocol include analyzing request response time, success rate, and error rate; KPIs related to the DNS protocol include evaluating query response time and domain name resolution success rate; and KPIs related to the FTP protocol include monitoring file transfer speed, success rate, and error messages.

[0082] Step 604: Evaluate the health status of the target application service based on the key performance indicators to obtain the application evaluation results.

[0083] In one embodiment, based on the key performance indicators extracted in step 603, combined with a preset evaluation model or threshold rules, the operating status of the target application service is comprehensively evaluated to determine whether it is in a normal, abnormal, or warning state, and finally an application health status evaluation result is generated.

[0084] After completing real-time and in-depth analysis, the data processing module can be used to further configure strategies and present results based on the real-time and in-depth analysis results. The data processing module includes an alarm module, a baseline module, and a reporting module.

[0085] The alarm module is used to monitor for anomalies in key business metrics. For example... Figure 7 As shown, alarm rules can be flexibly configured according to business needs, including selecting monitoring metrics (such as response latency, error rate, number of connection failures, etc.), setting threshold ranges, and setting trigger conditions (such as duration, fluctuation range). Historical data is stored in the database. Once a situation that meets the alarm rules is detected in the historical data, an alarm will be triggered, generating alarm information with detailed information, and stored in the database for further review by operations and maintenance personnel. In addition, relevant operations and maintenance personnel can be notified via email and SMS to achieve timely response to abnormal events.

[0086] The baseline module is used to set benchmark values ​​for key performance indicators to support subsequent anomaly detection and status assessment. For example... Figure 8As shown, based on historical data in the database, statistical analysis is performed on the historical data according to preset dimensions to calculate baseline values ​​reflecting normal behavior patterns, and the calculated baseline value statistical results are stored in the database. There are two ways to set the baseline value: one is to dynamically generate the baseline value based on historical data analysis, automatically identifying the range of normal business behavior by statistically analyzing the trend of indicator changes over a period of time; the other allows users to manually configure the baseline value according to actual business needs, improving flexibility and applicability. When an indicator is detected to exceed the threshold set by the baseline, for example, the number of packets per second (pps) of IGMP protocol (Internet Group Management Protocol) packets is set not to exceed 10,000 pps. If pps exceeds 10,000 pps, it is considered abnormal. At this time, the packet and its context information are stored so that users can retrieve and view them later when needed, realizing the recording and tracing of abnormal events. Historical data is obtained from the database, and real-time analysis results and in-depth analysis results are integrated through the target timestamp of the target packet to obtain historical data.

[0087] The reporting module is used to display and output real-time and in-depth analysis results in a structured manner. For example... Figure 9 As shown, users can configure relevant parameters on the web interface, such as specifying a time period, protocol, and application data package. The system retrieves corresponding historical data from the database, performs statistical processing, and generates multi-dimensional visual reports. These reports can cover key performance indicator trends, anomaly statistics, and session behavior distribution. Furthermore, the visual reports support exporting to various formats (such as PDF, Excel, and CSV) and can be configured to generate scheduled tasks, meeting diverse needs for daily operations and maintenance, audit compliance, and reporting.

[0088] Through the coordinated work of the above three modules, not only can a comprehensive understanding of the operational status of services be achieved, but also actionable decision support can be provided for operations and maintenance personnel, thereby improving the stability and security of network and business systems.

[0089] Monitoring network and application performance requires not only effective baseline settings and anomaly detection mechanisms, but also an intuitive way to display various statistical results and analytical data. To help users better understand and utilize these analytical results, the data visualization module uses rich charts to present various statistical results, providing users with a multi-dimensional perspective on network performance, application performance, alarms, reports, baselines, and other content.

[0090] In one embodiment, the data visualization module can utilize a Vue-based front-end interface, leveraging the features of a modern front-end framework to provide users with a flexible and responsive user experience. The front-end communicates with the back-end via a RESTful API to obtain real-time data and historical records, dynamically displaying them in chart format. This design not only enhances the intuitiveness and ease of use of data visualization but also supports user-defined views and export functionality, meeting the needs of different scenarios. For example, users can view network traffic trends, application error rate changes, and historical alarm events over a specific time period, thereby gaining a comprehensive understanding of the system's operational status and potential problems.

[0091] Example 2: Based on the full flow analysis method provided in the foregoing embodiments, the present invention also provides an apparatus for implementing the full flow analysis of the above method, such as... Figure 10 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The device for full-flow analysis in this embodiment includes one or more processors 21 and a memory 22. Figure 10 Take a processor 21 as an example.

[0092] Processor 21 and memory 22 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0093] The memory 22 serves as a non-volatile computer-readable storage medium for the full flow analysis method. It can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the full flow analysis method in the foregoing embodiments. The processor 21 executes various functional applications and data processing of the full flow analysis device by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the full flow analysis method of the foregoing embodiments.

[0094] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they perform the full flow analysis method described in the foregoing embodiments, for example, the method described above. Figures 1-6 The steps shown.

[0096] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 10 One of the processors 21 can enable the one or more processors to execute the full flow analysis method in the foregoing embodiments, for example, to perform the above-described... Figures 1-6 The steps shown.

[0097] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0098] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A full flow analysis method, characterized in that, include: The full traffic data in the network environment is captured based on the DPDK framework, and the full traffic data is processed to obtain the processed full traffic data. Parse the processed full traffic data to obtain at least one original data packet; Establish a mapping relationship between each of the original data packets and its corresponding first index information; Each of the original data packets is analyzed in real time to obtain the real-time analysis results corresponding to each of the original data packets; Based on the analysis request and the mapping relationship, the corresponding target data packet is retrieved from at least one original data packet; Perform cross-time dimension behavioral analysis on the target data packet to obtain in-depth analysis results; By integrating the real-time analysis results and the in-depth analysis results corresponding to the target data packets, a network status analysis report is generated.

2. The full flow analysis method according to claim 1, characterized in that, The method further includes: All raw data packets are grouped into fixed time windows, and a base timestamp for each window is calculated. For each group, all the original data packets contained in the group are written consecutively into a packet file; the filename of the packet file includes the base timestamp. Create a corresponding first index entry for each raw data packet; For each group, the first index entry corresponding to the original data packet contained in the group is written into an original index file, and the file name of the original index file contains the basic timestamp. The base timestamp is converted to local time, and the local time and the storage path of the corresponding original index file are stored in the database as the first index information.

3. The full flow analysis method according to claim 2, characterized in that, The first index entry includes a first index value. A corresponding first index value is established for each original data packet, including: Extract the feature information from the original data packet; The feature information is converted into multiple corresponding decimal values, and the decimal values ​​are used as the first index value of the original data packet.

4. The full flow analysis method according to claim 2, characterized in that, The first index entry also includes the offset and length of the original data packet in the corresponding packet file, and the method further includes: Parse the time range conditions and protocol feature conditions in the analysis request; Obtain second index information that meets the time range conditions; Locate the corresponding target index file in the file system based on the second index information; The second index entry in the target index file is filtered to obtain the offset and length that match the protocol feature conditions; Based on the offset and length, the corresponding original data packet is read from the packet file, and the original data packet is encapsulated into a standard pcap format to obtain the target data packet.

5. The full flow analysis method according to claim 1, characterized in that, The deep analysis results include session records and field information, and the method further includes: The target data packet is subjected to protocol layer parsing to identify the protocol information of the link layer, network layer, and transport layer; Based on the protocol information, the sessions in the target data packet are classified and statistically analyzed to generate session records for dialogues at each level; Each frame of data in the target data packet is parsed to obtain field information containing the content of the target data packet.

6. The full flow analysis method according to claim 1, characterized in that, The real-time analysis results include traffic classification results, and the method further includes: Each of the original data packets is decoded layer by layer to obtain the protocol type and protocol information used by each original data packet at different layers; For each raw data packet, the protocol type and protocol information corresponding to each layer are combined to obtain the protocol stack combination corresponding to the raw data packet. Based on the aforementioned protocol stack combination, the traffic of the original data packets is classified to obtain the traffic classification results.

7. The full flow analysis method according to claim 1, characterized in that, The real-time analysis results also include application evaluation results, and the method further includes: Extract application layer data related to the target application service from the original data packet; Identify the protocol type used by the application layer data; Based on the protocol type, a corresponding protocol parser is selected, and behavioral analysis is performed on the application layer data to obtain key performance indicators that reflect the operational status of the target application business. The health status of the target application service is assessed based on the key performance indicators, and the application assessment results are obtained.

8. The full flow analysis method according to claim 1, characterized in that, The method further includes: Obtain the real-time analysis results corresponding to the target data packet, wherein the real-time analysis results include the target timestamp of the abnormal event; Obtain the deep analysis results corresponding to the target timestamp, the deep analysis results including the protocol violation field; Based on the target timestamp, abnormal events and protocol violation fields are integrated to generate a network status analysis report.

9. A full flow analysis device, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the full flow analysis method according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the full flow analysis method as described in any one of claims 1-8.

Images