A method, system and device for extracting network traffic data packets
By dynamically adjusting the packet extraction logic, the highest priority rules are matched according to the rule state of the data flow and downgraded to the low priority rules, which solves the problems of low flexibility and waste of resources in traditional technology, and improves the extraction of network traffic packets Accuracy and system adaptability.
Patent Information
- Application Number
- CN202411314241.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-20
AI Technical Summary
When traditional network traffic monitoring technology processes large number of network data flows, it has low flexibility, low accuracy and serious resource waste. Especially when multiple filtering rules match, it is difficult to quickly extract data packets that meet the needs.
By obtaining the stream file and its corresponding highest priority filtering rules, dynamically adjusting the packet extraction logic, prioritize matching of the highest priority rules according to the change of the rule state of the data flow, and downgrade to the low priority rules when mismatch, to improve Accuracy and system adaptability of data packet extraction to avoid waste of resources.
Improve the accuracy of packet extraction and system flexibility, avoid the waste of computing, storage and network resources, and ensure that data packets that meet the needs can be extracted in a timely manner when query rules are received.
Smart Images

Figure CN118827437B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network traffic monitoring, and in particular to a method, system and device for extracting network traffic data packets. Background Art
[0002] With the continuous expansion of the current network scale and the explosive growth of business traffic in the network, traditional network traffic monitoring and analysis technology faces huge challenges. In related technologies, filtering rules are set through the content of protocol metadata to extract the required data packet content in the network traffic, so as to filter out some messages that do not meet the requirements and reduce the processing pressure of the computer system. However, when the message storage volume is large and each network data flow may match multiple filtering rules, how to quickly extract all the data packet content in the network data flow that meets the requirements is a problem that needs to be solved at present. Summary of the invention
[0003] In view of this, the present invention proposes a network traffic data packet extraction method, system and computer equipment, which solve the problems of low flexibility, low accuracy and waste of resources in the data packet extraction process.
[0004] Based on the above purpose, an embodiment of the present invention provides a method for extracting network traffic data packets, which specifically includes the following steps:
[0005] In response to receiving the query rule, obtaining a first stream file in the to-be-processed files and its corresponding highest priority filtering rule;
[0006] Determining a rule state of the first stream file according to the first stream file and the highest priority filtering rule;
[0007] In response to the rule state being a first state, determining whether a filtering rule corresponding to the first state matches the query rule;
[0008] In response to the filtering rule matching the query rule, the data packets in the first flow file are stored in the output array, the next flow file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as the new first flow file and the highest priority filtering rule, and the step of determining the rule status of the first flow file according to the first flow file and the highest priority filtering rule is returned until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
[0009] In some implementations, the network traffic data packet extraction method further includes:
[0010] If the rule state is the second state, searching in a preset data table whether there is cache information corresponding to the first stream file;
[0011] In response to the presence of first cache information corresponding to the first stream file in the preset data table, deleting the first cache information in the preset data table, and determining whether a filtering rule corresponding to the first stream file in the second state matches the query rule;
[0012] In response to the filtering rule not matching the query rule, obtaining the next stream file in the to-be-processed file and its corresponding highest-priority filtering rule as a new first stream file and the highest-priority filtering rule, and returning to the step of determining the rule status of the first stream file according to the first stream file and the highest-priority filtering rule;
[0013] In response to the filtering rule matching the query rule, the data packets in the first flow file are stored in the output array, the next flow file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as the new first flow file and the highest priority filtering rule, and the step of determining the rule status of the first flow file according to the first flow file and the highest priority filtering rule is returned until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
[0014] In some implementations, the network traffic data packet extraction method further includes:
[0015] If the rule state is the third state, the data flow identifier of the first stream file is used as a keyword, and the rule identifier or file sequence number of the first stream file is used as a key value;
[0016] Constructing target cache information based on the keyword and the key value and storing the target cache information in the preset data table;
[0017] The next stream file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as a new first stream file and the highest priority filtering rule, and the step of determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule is returned.
[0018] In some implementations, the network traffic data packet extraction method further includes:
[0019] monitoring whether the cache duration of the target cache information exceeds a first preset duration,
[0020] In response to the cache duration of the target cache information exceeding a first preset duration, the target cache information is deleted from the preset data table, and the data packets in the first stream file are stored in an output array.
[0021] In some implementations, the network traffic data packet extraction method further includes:
[0022] In response to the filtering rule not matching the query rule, the next stream file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as a new first stream file and the highest priority filtering rule, and the step of determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule is returned.
[0023] In some implementations, the step of determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule includes:
[0024] Determine whether all data packets in the first stream file meet their corresponding highest priority filtering rules;
[0025] In response to all data packets in the first stream file not satisfying the highest priority filtering rule of the first stream file, determining that the rule state of the first stream file is a third state, and detecting whether a new data packet in the network traffic is captured within a second preset time period;
[0026] In response to capturing a new data packet in the network traffic within a second preset time period, taking the new data packet as a first data packet, and detecting whether the first data packet satisfies the highest priority filtering rule;
[0027] In response to the first data packet satisfying the highest priority filtering rule, determining that the rule state of the first stream file is the first state, and determining that the filtering rule corresponding to the first state is the highest priority filtering rule;
[0028] In response to the first data packet not satisfying the highest priority filtering rule, the rule state of the first stream file is determined to be a second state, and the filtering rule corresponding to the second state is determined to be a next priority filtering rule of the highest priority filtering rule.
[0029] In some implementations, the network traffic data packet extraction method further includes:
[0030] In response to the absence of first cache information corresponding to the first stream file in the preset data table, determining whether a filtering rule corresponding to the second state matches the query rule;
[0031] In response to the filtering rule not matching the query rule, obtaining the next stream file in the to-be-processed file and its corresponding highest-priority filtering rule as a new first stream file and the highest-priority filtering rule, and returning to the step of determining the rule status of the first stream file according to the first stream file and the highest-priority filtering rule;
[0032] In response to the filtering rule matching the query rule, the data packets in the first flow file are stored in the output array, the next flow file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as the new first flow file and the highest priority filtering rule, and the step of determining the rule status of the first flow file according to the first flow file and the highest priority filtering rule is returned until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
[0033] In some implementations, the step of obtaining a first stream file from among the files to be processed includes:
[0034] According to the sequence of the file sequence numbers of the stream files in the to-be-processed file, the first stream file in the to-be-processed file is obtained as the first stream file.
[0035] Another aspect of the embodiment of the present invention further provides a network traffic data packet extraction system, comprising:
[0036] An acquisition module configured to acquire a first stream file and a corresponding highest priority filtering rule from among the files to be processed in response to receiving a query rule;
[0037] a determination module configured to determine a rule state of the first stream file according to the first stream file and the highest priority filtering rule;
[0038] a judgment module, configured to judge, in response to the rule state being a first state, whether a filtering rule corresponding to the first state matches the query rule;
[0039] A query module is configured to store the data packets in the first stream file into an output array in response to the filtering rule matching the query rule, obtain the next stream file in the to-be-processed file and the corresponding highest priority filtering rule as a new first stream file and the highest priority filtering rule, and return to the execution step of the determination module configuration until all the data packets in the output array are used as query results corresponding to the query rule after the query time ends.
[0040] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: at least one processor; and a memory, wherein the memory stores a computer program executable on the processor, and the computer program implements the steps of the above method when executed by the processor.
[0041] The present invention has at least the following beneficial technical effects: the network traffic data packet extraction method of the present invention determines the rule state change of the data flow when matching the rule according to the flow file and its corresponding highest priority filtering rule, which is helpful to better manage and process the data flow, and can dynamically adjust the data packet extraction logic according to the rule state of the data flow, to ensure that when the query rule is received, the data packet can be extracted according to the highest priority filtering rule matched by the data flow in priority, and when all data packets in the data flow do not meet the high priority filtering rule, the highest priority filtering rule is automatically downgraded to other low priority filtering rules to match the query rule by changing the rule state, and the data packet that meets the demand is extracted in time, which improves the adaptability and flexibility of the system, avoids the waste of computing resources, storage resources or network bandwidth, and adjusts the matching filtering rules according to the actual behavior of the data flow, which can improve the accuracy of data packet extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.
[0043] Figure 1 A block diagram of an embodiment of a method for extracting network traffic data packets provided by the present invention;
[0044] Figure 2 A schematic diagram of a flow chart of an embodiment of a process for extracting network traffic data packets provided by the present invention;
[0045] Figure 3 A schematic diagram of an embodiment of a network traffic data packet extraction system provided by the present invention;
[0046] Figure 4 A schematic structural diagram of an embodiment of a computer device provided by the present invention. DETAILED DESCRIPTION
[0047] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0048] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.
[0049] In the related art, a flow file and an index file are established to store data streams in a packet-by-packet storage manner. That is, a network capture tool is used to capture data packets in network traffic, and each data packet is parsed to obtain its five-tuple information, and data packets with the same five-tuple information are classified as a data stream. The flow file is used to store the five-tuple information and relative sequence number corresponding to the data stream. An entry is created for each data stream in the index file, and each entry contains the relative sequence number of the data stream to which each data packet belongs. The five-tuple information includes the source IP (Internet Protocol), destination IP, source port, destination port, and protocol type. When querying a data packet, the query rule is first parsed into a five-tuple form, and then the matching data stream is searched in the flow file according to the five-tuple information obtained by the parsing, and then the relative sequence number of the data stream in the index file is traversed to determine whether it is hit. If a data packet is hit, the message of the data packet is extracted from the message storage file according to the information in the index file.
[0050] The above-mentioned related technologies have the following defects:
[0051] 1. When a data stream matches multiple filtering rules and there is a priority between the filtering rules, if the query rule matches a filtering rule with a lower priority in the data stream, the data packets in the data stream will be extracted as the query result, resulting in inaccurate query results.
[0052] 2. When the data packets captured in the early stage of a data stream are identified as matching the first filtering rule, as the number of messages increases, the data packets captured later are identified as matching the second filtering rule. If the query rule matches the first filtering rule in the data stream at this time, all the data packets in the data stream will be extracted as the query result, resulting in incompletely correct query results.
[0053] In order to solve the above defects, the first aspect of the embodiments of the present invention provides an embodiment of a method for extracting network traffic data packets. Figure 1 As shown, it includes the following steps:
[0054] Step S100, in response to receiving the query rule, obtaining the first stream file in the to-be-processed files and its corresponding highest priority filtering rule.
[0055] In some embodiments, the query rules include filtering conditions or security policies input by the user. When the query rules are received, the pf (packet filtering) file is processed cyclically, and the pf file records the information of all data flows. According to the order of the file numbers of each data flow in the pf file, the flow files of the data flows that need to be processed currently, that is, the first flow files, are determined in sequence. For example, when the query rules are just received, the flow file of the data flow with the file number of the first sequence number is used as the first flow file. After the processing of the first flow file is completed, the flow file of the data flow with the file number of the second sequence number is processed in sequence and used as the first flow file. After the processing of the flow file of the last data flow recorded in the pf file is completed, the flow file of the data flow with the file number of the first sequence number is used as the first flow file again.
[0056] Step S200: determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule.
[0057] In some embodiments, it is determined whether all traffic packets in the first stream file meet the highest priority filtering rule to determine the rule state of the first stream file, and the rule state is recorded in the first stream file in the form of a drift FLAG (flag). The rule state includes a no-rule drift state, a rule drifting state, and a rule drifting end state. The no-rule drift state indicates that the data stream matches the same filtering rule from beginning to end, the rule drifting state indicates that the filtering rule of the data stream has changed during the matching process, and the drifting end indicates that the data stream has stably matched the new filtering rule after experiencing the rule change. Each rule state corresponds to a rule drift FLAG, which can be a simple string, enumeration value, integer value, or other type of variable, indicating different rule states. The drift FLAG takes -1, 0, and 1 as examples. When the drift FLAG is -1, the rule state of the first stream file is determined to be a no-rule drift state, when the drift FLAG is 0, the rule state of the first stream file is determined to be a rule drifting state, and when the drift FLAG is 1, the rule state of the first stream file is determined to be a rule drifting end state.
[0058] Step S300: In response to the rule state being the first state, it is determined whether the filtering rule corresponding to the first state matches the query rule.
[0059] In some embodiments, the first state is a no-regulation drift state.
[0060] Step S400, in response to the filtering rule matching the query rule, the data packets in the first stream file are stored in the output array, the next stream file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as the new first stream file and the highest priority filtering rule, and the step of determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule is returned until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
[0061] In some embodiments, Figure 2 The flowchart of an embodiment of the network traffic data packet extraction process provided by the present invention is shown. When the rule state of the currently processed data stream is determined to be a state without rule drift according to the rule drift FLAG, the filtering rule corresponding to the first stream file is the highest priority filtering rule at this time, and the rule identifier of the highest priority filtering rule of the data stream is directly matched with the rule identifier of the query rule. If the rule identifiers are consistent, the file sequence number corresponding to the data stream is transmitted to the output array, and the message of the data packet of the data stream is output as the query result, and the cache information of the data stream is cleared in the pf file. If they are inconsistent, the next data stream in the pf file is processed according to the file sequence number.
[0062] In some embodiments, if the currently processed data flow rule state is in the rule drifting state, the data flow identifier of the data flow is used as a keyword, and the file sequence number or rule identifier of the data flow is used as a key value to construct cache information, and the cache information is stored in the corresponding map to be output. If the currently processed data flow rule state is the rule drifting end state, the map to be output is searched for cache information corresponding to the data flow identifier of the data flow, if so, the file sequence number in the cache information is deleted in the map to be output, and the rule identifier of the filtering rule corresponding to the data flow in the rule drifting end state is matched with the rule identifier of the query rule, if the rule identifiers are consistent, the file sequence number corresponding to the data flow is transmitted to the output array, the message of the data packet of the data flow is output as the query result, and the cache information of the data flow is cleared in the pf file, if not consistent, the next data flow in the pf file is processed according to the file sequence number, if there is no cache information corresponding to the data flow identifier of the data flow in the map to be output, it is only determined whether the rule identifier of the filtering rule corresponding to the data flow in the rule drifting end state matches the rule identifier of the query rule. Figure 2 Here, ruleId is the rule identifier, flowIndex is the file number, that is, the number of the data flow in the pf file, and flowId is the data flow identifier, which is a global index of the data flow and is used to identify the number of the data flow in the entire message storage cycle.
[0063] The network traffic data packet extraction method of the present invention determines the rule state change of the data flow when matching the rule according to the flow file and its corresponding highest priority filtering rule, which is helpful to better manage and process the data flow, and can dynamically adjust the data packet extraction logic according to the rule state of the data flow, to ensure that when the query rule is received, the data packet can be extracted in priority according to the highest priority filtering rule matched by the data flow, and when all data packets in the data flow do not meet the high priority filtering rule, the highest priority filtering rule is automatically downgraded to other low priority filtering rules to match the query rule by changing the rule state, and the data packet that meets the demand is extracted in time, which improves the adaptability and flexibility of the system, avoids the waste of computing resources, storage resources or network bandwidth, and adjusts the matching filtering rules according to the actual behavior of the data flow, which can improve the accuracy of data packet extraction.
[0064] In some embodiments, the network traffic data packet extraction method of the present invention also includes: if the rule state is the second state, searching in the preset data table whether there is cache information corresponding to the first stream file; in response to the presence of first cache information corresponding to the first stream file in the preset data table, deleting the first cache information in the preset data table, and judging whether the filtering rule corresponding to the first stream file in the second state matches the query rule; in response to the filtering rule not matching the query rule, obtaining the next stream file in the file to be processed and its corresponding highest priority filtering rule as the new first stream file and the highest priority filtering rule, and returning to the step of determining the rule state of the first stream file based on the first stream file and the highest priority filtering rule; in response to the filtering rule matching the query rule, storing the data packet in the first stream file into the output array, obtaining the next stream file in the file to be processed and its corresponding highest priority filtering rule as the new first stream file and the highest priority filtering rule, and returning to the step of determining the rule state of the first stream file based on the first stream file and the highest priority filtering rule, until all the data packets in the output array are used as the query results corresponding to the query rules after the query time ends.
[0065] In some embodiments, the second state is the rule drift end state. The preset data table is a pre-created map to be output. The preset data table uses the data flow identifier of the data flow as the keyword field, and the file sequence number of the flow file or the rule identifier of the filtering rule as the key value field, and is used to cache the data flow to be output in the rule drift state. The first cache information includes the data flow identifier, the file sequence number and / or the rule identifier. The data flow identifier is used to uniquely identify a data flow, and can represent the sequence number of the data flow during the entire data packet message storage period, which helps the system track, identify and manage the data flow. The rule identifier is used to uniquely identify a filtering rule.
[0066] In some implementations, it is searched in all preset data tables whether there is cache information matching the data stream identifier corresponding to the first stream file. If so, the file sequence number in the cache information is deleted from the map to be output, and then it is determined whether the rule identifier of the filtering rule corresponding to the first stream file in the rule drift end state matches the rule identifier of the query rule. If not, it is directly determined whether the rule identifier of the filtering rule corresponding to the first stream file in the rule drift end state matches the rule identifier of the query rule.
[0067] In some embodiments, if the rule identifier of the filtering rule corresponding to the first stream file in the rule drift end state matches the rule identifier of the query rule, the file sequence number corresponding to the first stream file and the corresponding file sequence number in the cache information are added to the pre-created output array, and then the message of the data packet of the data stream is extracted as the query result, that is, the message of the data packet is extracted and output in the message storage file according to each file sequence number in the output array. In order to save space, the data packet can be compressed or only the key fields can be stored, and the message of the data packet can be stored in another file (or database), which can be called a message storage file. When storing the data packet, add the rule drift FLAG, rule identifier and data flow identifier in the flow file.
[0068] In some embodiments, the network traffic data packet extraction method of the present invention also includes: if the rule state is the third state, using the data flow identifier of the first stream file as the keyword, and the rule identifier or file sequence number of the first stream file as the key value; constructing the target cache information based on the keyword and the key value and storing the target cache information in a preset data table; obtaining the next stream file in the file to be processed and its corresponding highest priority filtering rule as the new first stream file and the highest priority filtering rule, and returning to the step of determining the rule state of the first stream file based on the first stream file and the highest priority filtering rule.
[0069] In some embodiments, the network traffic data packet extraction method of the present invention also includes: monitoring whether the cache duration of the target cache information exceeds a first preset duration, in response to the cache duration of the target cache information exceeding the first preset duration, deleting the target cache information in the preset data table, and storing the data packets in the first stream file into the output array.
[0070] In some embodiments, the third state is a rule drifting state. When it is determined that the rule state of the first stream file is a rule drifting state, the corresponding cache information is cached in the map to be output, so that after the drift of the data stream corresponding to the first stream file ends, the data stream is output according to the rule matching result. The longest time required for rule drift is set as the first preset time length. When the cache information is cached in the map to be output for more than the first preset time length, the data packet of the data stream corresponding to the cache information is written into the output array, and the cache information is deleted in the map to be output. The present invention adds a drift FLAG to the stream file to identify the rule state, and can quickly determine the data stream that meets the query rule and the highest priority filtering rule matches the query rule according to the rule state during the rule query, and extract the data packet message of the data stream in time. And cache the data streams with different filtering rules obtained by identifying different parts of the data packets to avoid extracting all the data packets in the data stream as the query result when a certain part of the filtering rule matches the query rule, resulting in incompletely correct query results. The present invention can improve the accuracy of system processing by being able to adjust the processing logic according to the actual behavior of the data stream. When the behavior of the data stream does not conform to the expected pattern of the high-priority filtering rule, the filtering rule can also be downgraded to other low-priority filtering rules through rule drift to match the query rule, thereby improving the adaptability and flexibility of the system.
[0071] In an example, it is assumed that a data flow matches two filtering rules, Rule 1 and Rule 2, where the priority of Rule 1 is higher than the priority of Rule 2. Rule 1 is shown in the following Table 1, and Rule 2 is shown in Table 2.
[0072] Table 1
[0073]
[0074] Table 2
[0075]
[0076] Continuing with the above example, the five-tuple of rule 1 and rule 2 are consistent, but rule 1 requires that the data flow message include FTP request or response content. When there is a message of three data packets, namely, syn packet, syn+ack packet and ack packet, in the data flow, and all of them meet the five-tuple in rules 1 and 2, since rule 1 has a high priority, the data flow will initially identify the filtering rule as rule 1, but the above three messages do not include FTP request or response content, wait for subsequent messages within the preset time of the timer and determine whether the subsequent messages include FTP request or response. At this time, the rule status of the data flow is the rule drifting state. If the timer times out and no other messages including FTP request or response content are received subsequently, then after the timeout, the filtering rule of the data flow including the above three messages will be identified as rule 2, and the rule status is the rule drift end state. If other messages including FTP request or response content are received before the timer times out, then the filtering rule will continue to be identified as rule 1, and the rule status is the rule drifting state.
[0077] In some embodiments, the network traffic data packet extraction method of the present invention also includes: in response to the filtering rule not matching the query rule, obtaining the next flow file in the file to be processed and its corresponding highest priority filtering rule as a new first flow file and the highest priority filtering rule, and returning to the step of determining the rule status of the first flow file based on the first flow file and the highest priority filtering rule.
[0078] In some embodiments, the step of determining the rule state of the first stream file according to the first stream file and the highest priority filtering rule includes: judging whether all data packets in the first stream file satisfy their corresponding highest priority filtering rule; in response to all data packets in the first stream file not satisfying the highest priority filtering rule of the first stream file, determining that the rule state of the first stream file is a third state, and detecting whether a new data packet in the network traffic is captured within a second preset time length; in response to a new data packet in the network traffic being captured within the second preset time length, taking the new data packet as the first data packet, and detecting whether the first data packet satisfies the highest priority filtering rule; in response to the first data packet satisfying the highest priority filtering rule, determining that the rule state of the first stream file is a first state, and determining that the filtering rule corresponding to the first state is the highest priority filtering rule; in response to the first data packet not satisfying the highest priority filtering rule, determining that the rule state of the first stream file is a second state, and determining that the filtering rule corresponding to the second state is the next priority filtering rule of the highest priority filtering rule.
[0079] In some embodiments, the network traffic data packet extraction method of the present invention also includes: in response to the first cache information corresponding to the first stream file not existing in the preset data table, determining whether the filtering rule corresponding to the second state matches the query rule; in response to the filtering rule not matching the query rule, obtaining the next stream file in the file to be processed and its corresponding highest priority filtering rule as the new first stream file and the highest priority filtering rule, and returning to the step of determining the rule state of the first stream file based on the first stream file and the highest priority filtering rule; in response to the filtering rule matching the query rule, storing the data packet in the first stream file into the output array, obtaining the next stream file in the file to be processed and its corresponding highest priority filtering rule as the new first stream file and the highest priority filtering rule, and returning to the step of determining the rule state of the first stream file based on the first stream file and the highest priority filtering rule, until the query time ends and all the data packets in the output array are used as the query results corresponding to the query rules.
[0080] In some embodiments, for each data stream, several filtering rules are identified based on its five-tuple, and the filtering rule with the highest priority is identified as the filtering rule of the data stream. When it is detected that all data packets in the data stream do not conform to the expected pattern of the filtering rule with the highest priority, a timer is started to monitor whether a message conforming to the expected pattern of the filtering rule with the highest priority is received before the timer times out (i.e., the second preset duration). At this time, the rule status of the data stream is a rule drifting state. If received, the highest priority filtering rule continues to be used as the filtering rule for the data stream. At this time, the rule status of the data stream is a rule drifting state. If not received, the next priority filtering rule identified is used as the filtering rule for the data stream. At this time, the rule status of the data stream is a rule drifting end state. The present invention can realize the extraction and query of intentions. Figure 1 The data packets of the data flow that are consistent and match the highest priority filtering rule are re-evaluated according to the new packets in the data flow, and all identified filtering rules and state change logs are recorded. The second preset time can be set according to the actual needs of the application and is not specifically limited here.
[0081] In some implementations, the step of obtaining a first stream file in the file to be processed includes: obtaining the first stream file in the file to be processed as the first stream file according to the sequence of file sequence numbers of the stream files in the file to be processed.
[0082] Based on the same inventive concept, according to another aspect of the present invention, Figure 3 As shown, an embodiment of the present invention further provides a network traffic data packet extraction system, comprising:
[0083] An acquisition module 110 is configured to acquire a first stream file in the to-be-processed file and its corresponding highest priority filtering rule in response to receiving a query rule;
[0084] A determination module 120, configured to determine a rule state of the first stream file according to the first stream file and the highest priority filtering rule;
[0085] The judgment module 130 is configured to judge whether the filtering rule corresponding to the first state matches the query rule in response to the rule state being the first state;
[0086] The query module 140 is configured to store the data packets in the first stream file into the output array in response to the matching of the filtering rule with the query rule, obtain the next stream file in the to-be-processed file and the corresponding highest priority filtering rule as the new first stream file and the highest priority filtering rule, and return to the execution step of the determination module configuration until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
[0087] The network traffic data packet extraction system of the present invention determines the rule state change of the data flow when matching the rule according to the flow file and its corresponding highest priority filtering rule, which is helpful to better manage and process the data flow, and can dynamically adjust the data packet extraction logic according to the rule state of the data flow, to ensure that when the query rule is received, the data packet can be extracted in priority according to the highest priority filtering rule matched by the data flow, and when all data packets in the data flow do not meet the expected state of the high priority filtering rule, the highest priority filtering rule is automatically downgraded to other low priority filtering rules to match the query rule by changing the rule state, and the data packet that meets the demand is extracted in time, thereby improving the adaptability and flexibility of the system, avoiding the waste of computing resources, storage resources or network bandwidth, and adjusting the matching filtering rules according to the actual behavior of the data flow, which can improve the accuracy of data packet extraction.
[0088] Based on the same inventive concept, according to another aspect of the present invention, Figure 4 As shown, an embodiment of the present invention further provides a computer device 30, which includes a processor 310 and a memory 320. The memory 320 stores a computer program 321 that can be run on the processor. When the processor 310 executes the program, the steps of the above method are performed.
[0089] Finally, it should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program can be a disk, an optical disk, a read-only storage memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding above-mentioned any method embodiments.
[0090] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.
[0091] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.
[0092] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.
[0093] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.
Claims
1. A method for extracting network traffic data packets, characterized in that: include In response to receiving the query rule, obtaining a first stream file in the to-be-processed files and its corresponding highest priority filtering rule; Determine the rule state of the first stream file according to the first stream file and the highest priority filtering rule, wherein the rule state includes a no-rule-drift state, a rule-drifting state, and a rule-drifting end state, wherein the no-rule-drift state indicates that the data stream matches the same filtering rule from beginning to end, the rule-drifting state indicates that the filtering rule of the data stream has changed during the matching process, and the drifting end indicates that the data stream has stably matched the new filtering rule after experiencing the rule change, the no-rule-drift state is the first state, the rule-drifting end state is the second state, and the rule-drifting state is the third state; If the rule state is the first state, determining whether the filtering rule corresponding to the first state matches the query rule; In response to the filtering rule matching the query rule, storing the data packets in the first flow file into the output array, obtaining the next flow file in the to-be-processed file and its corresponding highest priority filtering rule as a new first flow file and the highest priority filtering rule, and returning to the step of determining the rule state of the first flow file according to the first flow file and the highest priority filtering rule, until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends; If the rule state is the second state, searching in a preset data table whether there is cache information corresponding to the first stream file; In response to the presence of first cache information corresponding to the first stream file in the preset data table, deleting the first cache information in the preset data table, and determining whether a filtering rule corresponding to the first stream file in the second state matches the query rule; In response to the filtering rule not matching the query rule, obtaining the next stream file in the to-be-processed file and its corresponding highest-priority filtering rule as a new first stream file and the highest-priority filtering rule, and returning to the step of determining the rule status of the first stream file according to the first stream file and the highest-priority filtering rule; In response to the filtering rule matching the query rule, the data packets in the first flow file are stored in the output array, the next flow file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as the new first flow file and the highest priority filtering rule, and the step of determining the rule status of the first flow file according to the first flow file and the highest priority filtering rule is returned until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
2. The method for extracting network traffic data packets according to claim 1, characterized in that: Also includes: If the rule state is the third state, the data flow identifier of the first stream file is used as a keyword, and the rule identifier or file sequence number of the first stream file is used as a key value; Constructing target cache information based on the keyword and the key value and storing the target cache information in the preset data table; The next stream file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as a new first stream file and the highest priority filtering rule, and the step of determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule is returned.
3. The method for extracting network traffic data packets according to claim 2, characterized in that: Also includes: monitoring whether the cache duration of the target cache information exceeds a first preset duration, In response to the cache duration of the target cache information exceeding a first preset duration, the target cache information is deleted from the preset data table, and the data packets in the first stream file are stored in an output array.
4. The method for extracting network traffic data packets according to claim 1, characterized in that: Also includes: In response to the filtering rule not matching the query rule in the first state, the next stream file in the file to be processed and its corresponding highest priority filtering rule are obtained as the new first stream file and the highest priority filtering rule, and the step of determining the rule state of the first stream file based on the first stream file and the highest priority filtering rule is returned.
5. The method for extracting network traffic data packets according to claim 1, characterized in that: The step of determining the rule status of the first stream file according to the first stream file and the highest priority filtering rule comprises: Determine whether all data packets in the first stream file meet their corresponding highest priority filtering rules; In response to all data packets in the first stream file not satisfying the highest priority filtering rule of the first stream file, determining that the rule state of the first stream file is the third state, and detecting whether a new data packet in the network traffic is captured within a second preset time period; In response to capturing a new data packet in the network traffic within a second preset time period, taking the new data packet as a first data packet, and detecting whether the first data packet satisfies the highest priority filtering rule; In response to the first data packet satisfying the highest priority filtering rule, determining that the rule state of the first stream file is the first state, and determining that the filtering rule corresponding to the first state is the highest priority filtering rule; In response to the first data packet not satisfying the highest priority filtering rule, the rule state of the first stream file is determined to be the second state, and the filtering rule corresponding to the second state is determined to be the next priority filtering rule of the highest priority filtering rule.
6. The method for extracting network traffic data packets according to claim 1, characterized in that: Also includes: In response to the absence of first cache information corresponding to the first stream file in the preset data table, determining whether a filtering rule corresponding to the second state matches the query rule; In response to the filtering rule not matching the query rule, obtaining the next stream file in the to-be-processed file and its corresponding highest-priority filtering rule as a new first stream file and the highest-priority filtering rule, and returning to the step of determining the rule status of the first stream file according to the first stream file and the highest-priority filtering rule; In response to the filtering rule matching the query rule, the data packets in the first flow file are stored in the output array, the next flow file in the to-be-processed file and its corresponding highest priority filtering rule are obtained as the new first flow file and the highest priority filtering rule, and the step of determining the rule status of the first flow file according to the first flow file and the highest priority filtering rule is returned until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
7. The method for extracting network traffic data packets according to claim 1, characterized in that: The step of obtaining the first stream file in the file to be processed comprises: According to the sequence of the file sequence numbers of the stream files in the to-be-processed file, the first stream file in the to-be-processed file is obtained as the first stream file.
8. A network traffic data packet extraction system, characterized in that: include: An acquisition module configured to acquire a first stream file and a corresponding highest priority filtering rule from among the files to be processed in response to receiving a query rule; a determination module configured to determine a rule state of the first stream file according to the first stream file and the highest priority filtering rule, wherein the rule state includes a no-rule-drift state, a rule-drifting state, and a rule-drifting end state, wherein the no-rule-drift state indicates that the data stream matches the same filtering rule from beginning to end, the rule-drifting state indicates that the filtering rule of the data stream has changed during the matching process, and the drifting end indicates that the data stream has stably matched the new filtering rule after experiencing the rule change, the no-rule-drift state is the first state, the rule-drifting end state is the second state, and the rule-drifting state is the third state; a judgment module, configured to judge, in response to the rule state being the first state, whether a filtering rule corresponding to the first state matches the query rule; A query module configured to store the data packets in the first stream file into an output array if the filtering rule matches the query rule, obtain the next stream file in the to-be-processed file and the corresponding highest priority filtering rule as a new first stream file and the highest priority filtering rule, and return to the execution step configured by the determination module until all data packets in the output array are used as query results corresponding to the query rule after the query time ends; The query module is further configured to: if the rule state is the second state, search in the preset data table whether there is cache information corresponding to the first stream file; in response to the presence of first cache information corresponding to the first stream file in the preset data table, delete the first cache information in the preset data table, and determine whether the filtering rule corresponding to the first stream file in the second state matches the query rule; in response to the filtering rule not matching the query rule, obtain the next stream file in the to-be-processed file and its corresponding highest priority filtering rule as a new first stream file and the highest priority filtering rule, and return to the step of determining the rule state of the first stream file according to the first stream file and the highest priority filtering rule; in response to the filtering rule matching the query rule, store the data packet in the first stream file into the output array, obtain the next stream file in the to-be-processed file and its corresponding highest priority filtering rule as a new first stream file and the highest priority filtering rule, and return to the step of determining the rule state of the first stream file according to the first stream file and the highest priority filtering rule, until all the data packets in the output array are used as the query results corresponding to the query rule after the query time ends.
9. A computer device comprising: at least one processor; as well as A memory storing a computer program executable on the processor, wherein the processor executes the steps of the method according to any one of claims 1 to 7 when executing the program.
