Method and system for quickly searching network full-flow session flow
By establishing TCP session index and Bronze index in the full network traffic, the performance bottleneck problem caused by the discontinuity of the session flow in the full network traffic backtracking analysis is solved, the search speed of the session flow is improved, and the foundation for abnormal data mining and forensics is provided. .
Patent Information
- Application Number
- CN202411108649.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
AI Technical Summary
In network full traffic backtracking analysis, packets of the same session stream are usually discontinuous and disordered, resulting in the need to traverse all original traffic to extract session stream information that meets the conditions, affecting subsequent retrieval, extraction and analysis performance.
By establishing a TCP session index in the original traffic captured by the network card, and creating a Bronze index of IP and port based on this, we will prioritize whether the data of the specified IP or port exists in the Bronze index. Only when it exists will the session index be loaded. Extracts the session stream that meets the criteria.
It significantly improves the extraction speed of session streams, solves the performance bottleneck problem of session stream export in full traffic backtracking analysis, and provides a solid foundation for subsequent abnormal data mining, analysis and forensics.
Smart Images

Figure CN119011249A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication technology, and in particular to a method and system for quickly searching for a full-flow conversation flow on a network. Background Art
[0002] The core of the network full-process backtracking analysis system is to collect and store all the original traffic information. With the storage of the original traffic, the currently detected attack behavior can be associated with the historical traffic to achieve complete attack tracing and forensic analysis. However, when collecting and capturing network traffic messages, the messages of the same session flow are usually discontinuous and disordered. Therefore, it is necessary to traverse all the original traffic to extract the session flow information that meets the conditions, which greatly affects the performance of subsequent retrieval, extraction, and analysis.
[0003] At present, the method for storing full-flow conversation flow data in the network still adopts the real-time storage method: the real-time storage method is to store the original flow captured by the network card in files, establish a TCP session index through real-time analysis of the original flow, and store the five-tuple information of the TCP session (source IP, source port, destination IP, destination port, protocol name), the location and length information of all messages in the session, the storage file information to which the message belongs, etc. In this way, when extracting the conversation flow subsequently, it is necessary to judge the storage location and length information of the messages in the session that meet the conditions in the index file in turn, which affects the extraction speed. Summary of the invention
[0004] Based on this, the embodiment of the present application provides a method and system for quickly searching for full-flow session flows in a network. The present application will generate additional Bloom indexes of IP and port for the session indexes of the original flow and session message flow captured by the network card. In this way, when extracting data of a specified IP or port later, it is first determined whether it exists in the Bloom index. Only if it exists will the session index continue to be loaded and the specified flow be extracted, which greatly improves the extraction speed. It solves the performance bottleneck problem of session flow export in full-flow backtracking analysis.
[0005] In a first aspect, a method for quickly searching for a full-flow session flow in a network is provided, the method comprising:
[0006] Cache the original messages captured from network traffic into memory;
[0007] While caching the original message, the message is parsed to extract the five-tuple information to establish a TCP session index; wherein the five-tuple information includes the source IP, source port, destination IP, destination port, and protocol name;
[0008] Cache the TCP session index into the memory queue;
[0009] When the original message cache reaches the preset condition, the original messages of the same session are continuously written to the disk file according to the TCP session index order in the memory queue;
[0010] Create a Bloom index of IP and port based on the TCP session index;
[0011] The session is queried based on the five-tuple information to determine whether it hits the Bloom index. If it hits, the session stream index is loaded, and the sessions that meet the conditions are extracted for downloading.
[0012] Optionally, cache the original messages captured from the network traffic into memory, including:
[0013] Monitor network traffic through the network card and capture passing messages in real time;
[0014] The captured messages are immediately transferred to the memory, forming a temporary buffer area;
[0015] And during the caching process, monitor the memory usage.
[0016] Optionally, when the original message buffer reaches a preset condition, the original message buffer reaches a preset size limit or a collection time limit.
[0017] Optionally, original packets of the same session are continuously written to a disk file according to the TCP session index order in the memory queue, including:
[0018] According to the TCP session index order in the memory queue, TCP session indexes are obtained from the memory queue in sequence; each index includes five-tuple information of the session and the position and length information of the message in the cache;
[0019] Check the five-tuple information in the index to determine whether the message belongs to the same session;
[0020] Based on the message location and length information in the index, the system locates the specific message data in the memory;
[0021] Read the located message data from the memory and write it to the disk file in the order of the session;
[0022] After the message is written to disk, the message location information in the session index is updated.
[0023] Optionally, based on the TCP session index, create a Bloom index of IP and port, including:
[0024] Determine the hash function and bit array size of the Bloom index;
[0025] Extract the IP and port information of each session from the TCP session index;
[0026] Design multiple different hash functions to map IP and port information to different positions in the bit array;
[0027] Initialize the bit array with all bits in the initial state of 0. For each session's IP and port information, use the selected hash function to calculate its multiple positions in the bit array and set the bits at these positions to 1.
[0028] Optionally, querying the session according to the five-tuple information determines whether the Bloom index is hit, loading the session flow index when it is hit, and extracting the session that meets the condition for downloading, including:
[0029] According to the query requirements, determine the five-tuple information, use the Bloom index hash function to process the IP and port information in the five-tuple, and calculate the position in the Bloom index bit array;
[0030] For each position calculated by the hash function, check whether the corresponding bit in the Bloom index bit array is 1; when all the bits of the calculated positions are 1, it is considered that the session exists in the TCP session index; if any bit is 0, it can be determined that the session does not exist in the index;
[0031] If the Bloom index query hits, the session index is loaded. When the Bloom index query hits, the corresponding TCP session index is loaded according to the five-tuple information;
[0032] Use the loaded session index to locate the session data stored in the disk file according to the start and end position information of the session;
[0033] Based on the information in the index, extract the complete session data from the disk file.
[0034] In a second aspect, a network full-flow session flow fast search system is provided, the system comprising:
[0035] The capture module is used to cache the original messages captured from the network traffic into the memory;
[0036] Establishing a module for establishing a TCP session index while caching the original message;
[0037] A cache module is used to cache TCP session indexes into a memory queue;
[0038] The writing module is used to continuously write the original messages of the same session into the disk file according to the TCP session index order in the memory queue when the original message cache reaches a preset condition;
[0039] Index module, used to create Bloom index of IP and port based on TCP session index;
[0040] The query module is used to query the session according to the five-tuple information to determine whether it hits the Bloom index, load the session flow index when it hits, and extract the session that meets the conditions for downloading.
[0041] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for quickly searching for a full-traffic network session flow described in any one of the first aspects is implemented.
[0042] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for quickly searching for a full-traffic network session flow described in any one of the first aspects is implemented.
[0043] In a fifth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the method for quickly searching for network full-traffic session flows as described in any one of the first aspects above.
[0044] The embodiment of the present application first caches the original message captured from the network traffic into the memory; while caching the original message, establishes the TCP session index; then caches the TCP session index into the memory queue; when the original message cache reaches the preset condition, the original message of the same session is continuously written into the disk file according to the TCP session index order in the memory queue; finally, based on the TCP session index, creates the Bloom index of IP and port, queries the session according to the five-tuple information to determine whether the Bloom index is hit, loads the session flow index when it is hit, and extracts the session that meets the conditions for downloading. It can be seen that the beneficial effects brought by the provided technical solution include being able to effectively solve the performance bottleneck problem of session flow downloading in the full traffic backtracking analysis, and by establishing the Bloom index of the session flow and improving the search speed of the session flow, a solid foundation is established for subsequent abnormal data mining, analysis, and evidence collection. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0046] Figure 1 A flowchart of a method for quickly searching for a full-flow conversation flow on a network provided by an embodiment of the present application;
[0047] Figure 2 A flowchart provided for an optional embodiment of the present application;
[0048] Figure 3 A block diagram of a network full-flow conversation flow fast search system provided in an embodiment of the present application;
[0049] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0051] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.
[0052] The present invention discloses a method for quickly searching TCP / UDP session flows in network full-flow backtracking analysis applications, which solves the performance bottleneck problem of session flow export in full-flow backtracking analysis, facilitates the rapid extraction of original network session messages, and lays a solid foundation for subsequent abnormal data mining, analysis, and evidence collection.
[0053] Please refer to Figure 1 , which shows a flowchart of a method for quickly searching for a full-flow session flow in a network provided by an embodiment of the present application. The method may include the following steps:
[0054] S1 caches the original messages captured from the network traffic into memory.
[0055] In the embodiment of the present application, the original message captured from the network traffic is first cached in the memory. The purpose of this step is to ensure that the message data is continuous and complete, which is convenient for subsequent processing and analysis. During the caching process, the system needs to monitor the memory usage in real time to avoid memory overflow.
[0056] In the embodiment of the present application, the network traffic is monitored by the network card, and the passing messages are captured in real time;
[0057] The captured messages are immediately transferred to the memory, forming a temporary buffer area;
[0058] And during the caching process, monitor the memory usage.
[0059] S2, while caching the original message, parses the message and extracts the five-tuple information to establish a TCP session index.
[0060] The five-tuple information includes source IP, source port, destination IP, destination port, and protocol name. When caching messages, a TCP session index is established. The index information includes the five-tuple information (source IP, source port, destination IP, destination port, protocol name), the position and length information of each message in the session in the cached message, and the same five-tuple information indicates the same session;
[0061] S3, cache the TCP session index into the memory queue.
[0062] In the embodiment of the present application, the established TCP session index is then cached in a memory queue. The memory queue is used to manage the index data to ensure that they can be quickly accessed and processed in the order in which they were collected.
[0063] S4, when the original message cache reaches the preset condition, the original messages of the same session are continuously written to the disk file according to the TCP session index order in the memory queue.
[0064] When the original message cache reaches the set size or collection time, the TCP session indexes in the memory queue are obtained in sequence. According to the position and length information of each message in the same session in the cached message, the original messages of the same session are continuously written to the file, and the message position information of the session is modified to the start and end positions of the file.
[0065] In an embodiment of the present application, TCP session indexes are obtained from the memory queue in sequence according to the TCP session index order in the memory queue; wherein each index contains five-tuple information of the session and the position and length information of the message in the cache;
[0066] Check the five-tuple information in the index to determine whether the message belongs to the same session;
[0067] Based on the message location and length information in the index, the system locates the specific message data in the memory;
[0068] Read the located message data from the memory and write it to the disk file in the order of the session;
[0069] After the message is written to disk, the message location information in the session index is updated.
[0070] S5: Create a Bloom index of IP and port based on the TCP session index.
[0071] Based on the TCP session index, the system further creates a Bloom index of IP and port. A Bloom index is a probabilistic data structure used to quickly determine whether an element is in a set.
[0072] Through the Bloom index, the system can quickly determine whether the session of a specific IP and port exists in the index, thereby improving the speed of session search.
[0073] In an embodiment of the present application, a hash function and a size of a bit array of a Bloom index are determined;
[0074] Extract the IP and port information of each session from the TCP session index;
[0075] Design multiple different hash functions to map IP and port information to different positions in the bit array;
[0076] Initialize the bit array with all bits in the initial state of 0. For each session's IP and port information, use the selected hash function to calculate its multiple positions in the bit array and set the bits at these positions to 1.
[0077] S6, query the session according to the five-tuple information to determine whether it hits the Bloom index, load the session flow index if it hits, and extract the session that meets the conditions for downloading.
[0078] In the embodiment of the present application, according to the query requirements, the five-tuple information is determined, the IP and port information in the five-tuple is processed using the hash function of the Bloom index, and the position in the Bloom index bit array is calculated;
[0079] For each position calculated by the hash function, check whether the corresponding bit in the Bloom index bit array is 1; when all the bits of the calculated positions are 1, it is considered that the session exists in the TCP session index; if any bit is 0, it can be determined that the session does not exist in the index;
[0080] If the Bloom index query hits, the session index is loaded. When the Bloom index query hits, the corresponding TCP session index is loaded according to the five-tuple information;
[0081] Use the loaded session index to locate the session data stored in the disk file according to the start and end position information of the session;
[0082] Based on the information in the index, extract the complete session data from the disk file.
[0083] In an optional embodiment of the present application, a fast search method for TCP / UDP session flows in a network full-flow backtracking analysis application can be implemented by additionally establishing a Bloom index of IP and port based on the session index.
[0084] The process of searching for a session flow is as follows: Figure 2 As shown, the specific implementation process is as follows:
[0085] 1. Cache the original packets captured from the traffic into the memory;
[0086] 2. Create a TCP session index while caching the message. The index information includes five-tuple information (source IP, source port, destination IP, destination port, protocol name), the position and length information of each message in the session in the cache message. The same five-tuple information indicates the same session.
[0087] 3. Cache the TCP session index into the memory queue;
[0088] 4. When the original message cache reaches the set size or collection time, the TCP session indexes in the memory queue are obtained in sequence. According to the position and length information of each message in the same session in the cached message, the original messages of the same session are continuously written to the file, and the message position information of the session is modified to the start and end positions of the file.
[0089] 5. Based on the TCP session index, a Bloom index of IP and port is established, so that the IP and port can be quickly located in this session index;
[0090] 6. Subsequently, the session is queried based on the five-tuple information, and the IP and port of the session flow are prioritized to determine whether the Bloom index is hit. Only when the Bloom index is hit will the session flow index be loaded and the session that meets the conditions will be extracted for download.
[0091] In the embodiments of the present application, the innovative features specifically include:
[0092] 1) Original message cache and TCP session index cache
[0093] The message information of the TCP session flow is saved in a cache manner. When a trigger condition is met, the original message of the session flow is continuously written and the message start and end positions of the session flow are modified.
[0094] 2) TCP session stream index structure
[0095] Only the five-tuple information of the message, the start and end positions of the message, and the file to which the message belongs can be used for subsequent analysis and forensics, reducing the storage space of the TCP session index.
[0096] 3) Session stream download
[0097] The IP and port of the session flow are used to determine whether the Bloom index is hit. Only when the Bloom index is hit will the session flow index be loaded and the sessions that meet the conditions are extracted for download.
[0098] This application can effectively solve the performance bottleneck problem of session stream downloading in full-flow backtracking analysis, and lay a solid foundation for subsequent abnormal data mining, analysis, and evidence collection by establishing a Bloom index of the session stream and improving the search speed of the session stream.
[0099] In one embodiment of the present application, please refer to Figure 3 , and also provides a block diagram of a network full-flow conversation flow fast search system, the system comprising:
[0100] The capture module is used to cache the original messages captured from the network traffic into the memory;
[0101] Establishing a module for establishing a TCP session index while caching the original message;
[0102] A cache module is used to cache TCP session indexes into a memory queue;
[0103] The writing module is used to continuously write the original messages of the same session into the disk file according to the TCP session index order in the memory queue when the original message cache reaches a preset condition;
[0104] Index module, used to create Bloom index of IP and port based on TCP session index;
[0105] The query module is used to query the session according to the five-tuple information to determine whether it hits the Bloom index, load the session flow index when it hits, and extract the session that meets the conditions for downloading.
[0106] For the specific definition of the network full-flow conversation flow fast search system, please refer to the definition of the network full-flow conversation flow fast search method above, which will not be repeated here. Each module in the above-mentioned network full-flow conversation flow fast search system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0107] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 4As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to quickly search for data in the full-flow conversation flow of the network. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for quickly searching for the full-flow conversation flow of the network is implemented.
[0108] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for quickly searching for full-flow session flows in the network are implemented.
[0109] In one embodiment of the present application, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for quickly searching for full-traffic conversation flows in the network.
[0110] The computer-readable storage medium and computer program product provided in this embodiment have similar implementation principles and technical effects to those of the above method embodiments, and are not described in detail here.
[0111] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0112] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0113] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.
Claims
1. A method for quickly searching for full-flow conversation flows on a network, characterized in that: The method comprises: Cache the original messages captured from network traffic into memory; While caching the original message, the message is parsed to extract the five-tuple information to establish a TCP session index; wherein the five-tuple information includes the source IP, source port, destination IP, destination port, and protocol name; Cache the TCP session index into the memory queue; When the original message cache reaches the preset condition, the original messages of the same session are continuously written to the disk file according to the TCP session index order in the memory queue; Create a Bloom index of IP and port based on the TCP session index; The session is queried based on the five-tuple information to determine whether it hits the Bloom index. If it hits, the session stream index is loaded, and the sessions that meet the conditions are extracted for downloading.
2. The method for quickly searching for a full-flow network session flow according to claim 1, characterized in that: Cache the original messages captured from network traffic into memory, including: Monitor network traffic through the network card and capture passing messages in real time; The captured messages are immediately transferred to the memory, forming a temporary buffer area; And during the caching process, monitor the memory usage.
3. The method for quickly searching for a full-flow network session flow according to claim 1, characterized in that: When the original message buffer reaches a preset condition, including the original message buffer reaching a preset size limit or a collection time limit.
4. The method for quickly searching for a full-flow network session flow according to claim 1, characterized in that: The original packets of the same session are written to the disk file in sequence according to the TCP session index order in the memory queue, including: According to the TCP session index order in the memory queue, TCP session indexes are obtained from the memory queue in sequence; each index includes five-tuple information of the session and the position and length information of the message in the cache; Check the five-tuple information in the index to determine whether the message belongs to the same session; Based on the message location and length information in the index, the system locates the specific message data in the memory; Read the located message data from the memory and write it to the disk file in the order of the session; After the message is written to disk, the message location information in the session index is updated.
5. The method for quickly searching for a full-flow network session flow according to claim 1, characterized in that: Based on the TCP session index, create a Bloom index of IP and port, including: Determine the hash function and bit array size of the Bloom index; Extract the IP and port information of each session from the TCP session index; Design multiple different hash functions to map IP and port information to different positions in the bit array; Initialize the bit array with all bits in the initial state of 0. For each session's IP and port information, use the selected hash function to calculate its multiple positions in the bit array and set the bits at these positions to 1.
6. The method for quickly searching for a full-flow conversation flow of a network according to claim 1, characterized in that: The session is queried based on the five-tuple information to determine whether the Bloom index is hit. If it is hit, the session stream index is loaded, and the sessions that meet the conditions are extracted for download, including: According to the query requirements, determine the five-tuple information, use the Bloom index hash function to process the IP and port information in the five-tuple, and calculate the position in the Bloom index bit array; For each position calculated by the hash function, check whether the corresponding bit in the Bloom index bit array is 1; when all the bits of the calculated positions are 1, it is considered that the session exists in the TCP session index; if any bit is 0, it can be determined that the session does not exist in the index; If the Bloom index query hits, the session index is loaded. When the Bloom index query hits, the corresponding TCP session index is loaded according to the five-tuple information; Use the loaded session index to locate the session data stored in the disk file according to the start and end position information of the session; Based on the information in the index, extract the complete session data from the disk file.
7. A network full-flow conversation flow fast search system, characterized in that: The system comprises: The capture module is used to cache the original messages captured from the network traffic into the memory; Establishing a module for establishing a TCP session index while caching the original message; A cache module is used to cache TCP session indexes into a memory queue; The writing module is used to continuously write the original messages of the same session into the disk file according to the TCP session index order in the memory queue when the original message cache reaches a preset condition; Index module, used to create Bloom index of IP and port based on TCP session index; The query module is used to query the session according to the five-tuple information to determine whether it hits the Bloom index, load the session flow index when it hits, and extract the session that meets the conditions for downloading.
8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for quickly searching for a full-traffic conversation flow of a network as claimed in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for quickly searching for a full-traffic conversation flow of a network as described in any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for quickly searching for a full-traffic conversation flow on a network as described in any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Network traffic storage method, electronic equipment, storage medium and computer program product
CN119697292A
Hierarchical session management method and device based on FPGA
CN120281710A
