Message rule matching method and device and electronic equipment
By extracting and matching the characteristic data of the packets, the problem of rough matching of message detection rules and insufficient recognition accuracy in the prior art is solved, and high-precision and flexible message detection are realized to adapt to complex network environments.
Patent Information
- Application Number
- CN202411194580.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the rules for message detection are rough, the recognition accuracy is insufficient and the flexibility is poor, making it difficult to support complex network environments and diversified application scenarios.
By obtaining valid messages at the network exit, extracting their characteristic data, determining the filtering rule metadata field in the Baotou protocol stack, and matching them with the preset fields, obtaining the matching results to achieve refined message detection.
It improves the recognition accuracy and flexibility of message detection, and can more accurately identify and process network traffic, adapt to the needs of diversified network applications.
Smart Images

Figure CN118944958A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer communications, and in particular to a message rule matching method, device and electronic equipment. Background Art
[0002] As an advanced network monitoring strategy, Deep Packet Inspection (DPI) improves the traditional method of only checking the communication protocol header, and goes deep into the application layer to conduct detailed analysis and screening of the specific content of each data packet. This method expands the ability of traffic monitoring, enabling comprehensive insight into and control of every transmission detail in the network. DPI technology plays a key role in ensuring network security, implementing content filtering, and effectively managing network traffic. However, this technology has a high dependence on hardware performance because it has to handle a large amount of data parsing tasks. With the widespread adoption of broadband networks, the need for high-performance hardware has become more urgent to handle larger volumes of traffic.
[0003] In addition, the number of applications in the current network environment is increasing day by day, and each application often has unique detection specifications, which has led to a larger set of rules and higher computing requirements. To solve these problems, DPI systems are often equipped with high-performance processors and agile software designs, so that they can quickly adjust and adapt to new threats or changing application scenarios. Building and maintaining efficient DPI solutions is not only a process of technical deployment, but also requires continuous optimization to ensure high-speed operation and accurate analysis results. Such a system must have the ability to handle increasing data loads and the flexibility to adapt to diverse network applications. However, the rule matching for packet detection in related technologies is rough, and there are problems such as insufficient recognition accuracy and poor flexibility.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a message rule matching method, device and electronic device to at least solve the technical problems in the related art of rough rule matching for message detection, insufficient recognition accuracy and poor flexibility.
[0006] According to one aspect of an embodiment of the present application, a method for matching message rules is provided, including: obtaining a valid message at a network exit; extracting feature data of the valid message; determining a metadata field having a filtering rule in a packet header protocol stack of the valid message from the feature data, and splicing the metadata field; performing rule matching on the spliced metadata field and a preset field to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
[0007] Optionally, after obtaining a valid message at the network exit, the method also includes: obtaining five-tuple information of the valid message, wherein the five-tuple information includes source IP, destination IP, source address, destination address and transport layer protocol; determining a hash value corresponding to the valid message based on the five-tuple information; and determining an output path of the valid message based on the hash value.
[0008] Optionally, before extracting the characteristic data of the valid message, the method also includes: judging whether there is a valid message in the flow table record based on the quintuple information of the valid message to obtain a first judgment result; when the first judgment result indicates that there is no valid message in the flow table record, creating a new record for storing the valid message, and assigning an UNKNOWN label to the newly created flow table record corresponding to the valid message; when the first judgment result indicates that there is a message record corresponding to the valid message in the flow table record, judging whether the label of the message record is UNKNOWN to obtain a second judgment result; when the second judgment result is no, no longer identifying the valid message; when the second judgment result is yes or a new record is created to store the valid message, identifying the header protocol stack and load data of the valid message; determining the quintuple information, header protocol stack and load data of the valid message as the characteristic data of the valid message.
[0009] Optionally, determining a metadata field having a filtering rule in a packet header protocol stack of a valid message from characteristic data includes: determining a protocol type of the valid message from the packet header protocol stack; determining whether a transport layer where each protocol type is located carries payload data; and, when payload data is carried in the transport layer where each protocol type is located, parsing the corresponding protocol to determine the metadata field.
[0010] Optionally, the spliced metadata field and the preset field are subjected to rule matching to obtain a matching result, including: obtaining a payload in a valid message; obtaining an initial matching result based on the payload, the spliced metadata field and the preset field, wherein the initial matching result is used to represent the matching result of the secondary rule unit; determining a matching result based on the initial matching result, wherein the matching result is used to represent the matching result of the primary rule unit, and the primary rule unit is a parent rule unit of the secondary rule unit.
[0011] Optionally, an initial matching result is obtained based on the payload, the concatenated metadata field and the preset field, including: determining the payload as the first search mother string, and determining the concatenated metadata field as the second search mother string; determining the first search substring for payload matching from the preset field, and determining the second search substring for metadata matching; matching the first search mother string and the first search substring, and matching the second search mother string and the second search substring to obtain an initial matching result.
[0012] Optionally, determining the matching result based on the initial matching result includes: determining the id number of the secondary rule unit from the initial matching result; determining all parent rule units corresponding to the id number of the secondary rule unit from the multi-level rule unit storage table; and determining the parent rule unit as the matching result.
[0013] According to another aspect of an embodiment of the present application, a message rule matching device is also provided, including: an acquisition module, used to acquire valid messages at a network exit; an extraction module, used to extract characteristic data of the valid messages; a determination module, used to determine the metadata fields with filtering rules in the packet header protocol stack of the valid messages from the characteristic data, and to splice the metadata fields; a matching module, used to perform rule matching on the spliced metadata fields and preset fields to obtain a matching result, wherein the matching result includes retrieval rule attributes corresponding to the valid messages.
[0014] According to another aspect of the embodiment of the present application, an electronic device is also provided, including: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: obtaining a valid message at a network exit; extracting feature data of the valid message; determining a metadata field having a filtering rule in a packet header protocol stack of the valid message from the feature data, and splicing the metadata field; performing rule matching on the spliced metadata field and a preset field to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned message rule matching method by running the computer program.
[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned message rule matching method when executed by a processor.
[0017] In an embodiment of the present application, a valid message at a network exit is obtained; feature data of the valid message is extracted; a metadata field having a filtering rule in a packet header protocol stack of the valid message is determined from the feature data, and the metadata field is spliced; the spliced metadata field and a preset field are matched by rules to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message, thereby achieving the purpose of refined message detection, thereby realizing the technical effect of providing high message recognition accuracy, and further solving the technical problem in the related art that the rule matching for message detection is rough, the recognition accuracy is insufficient, and the flexibility is poor. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0019] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a message rule matching method according to an embodiment of the present application;
[0020] Figure 2 It is a flow chart of a message rule matching method according to an embodiment of the present application;
[0021] Figure 3 is a processing flow chart of a protocol metadata extraction unit according to an embodiment of the present application;
[0022] Figure 4 It is a structural diagram of a message rule matching device according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0025] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:
[0026] DPI (Deep Packet Inspection): Deep packet inspection technology adds application protocol identification, packet content detection and deep decoding of application layer data on top of traditional IP packet inspection technology (detection and analysis of packet elements contained between OSI L2-L4).
[0027] DPI in related technologies has the following limitations, such as too coarse rule matching, limited coverage, and difficulty in supporting complex regular expressions and rule combinations, resulting in problems such as insufficient recognition accuracy and poor flexibility during query. In order to solve the problems existing in related technologies, the present application embodiment provides a message rule matching method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.
[0028] The message rule matching method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a message rule matching method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0029] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0030] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the message rule matching method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned message rule matching method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0031] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0032] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .
[0033] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.
[0034] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a message rule matching method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0035] Figure 2 is a flow chart of a message rule matching method according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:
[0036] Step S202, obtaining a valid message at the network egress.
[0037] In the above step S202, valid messages refer to data packets that are correctly processed and forwarded at the network exit. These data packets usually have correct destination addresses and port numbers, comply with network protocol standards, and can be correctly routed and transmitted in the network. Obtaining valid messages at the network exit can help network administrators monitor network traffic and detect potential security issues. By analyzing valid messages, administrators can understand the communication situation in the network, discover abnormal activities in time, and take necessary measures to protect network security.
[0038] In an embodiment of the present application, the above step S202 can be implemented by a data acquisition unit, which can be deployed in the network in a serial or bypass manner, receive message data in the network, and filter out the valid part, and forward it to the subsequent process. The so-called valid message refers to a message with potential identification value in the scenarios of application identification, terminal identification or user identification in DPI identification. In a session, only the necessary part of the message is identified, and the unnecessary identification of the subsequent message is skipped under the condition that the previous message has been identified, which can greatly improve the overall performance of DPI and reduce the processing pressure of the subsequent units.
[0039] Step S204, extracting characteristic data of valid messages.
[0040] In the above step S204, extracting the characteristic data of valid messages refers to obtaining and analyzing the key attributes and information of valid messages from network communications. These characteristic data may include source address, destination address, port number, protocol type, transmission rate, data size, timestamp, etc. By extracting the characteristic data of valid messages, network administrators can better understand the patterns and trends of network traffic, identify abnormal behaviors in the network, and perform network performance optimization and security monitoring.
[0041] In an embodiment of the present application, step S204 can be implemented, for example, by a message processing unit, which includes two subunits, namely a message parsing unit and a flow association processing unit. The main function of the message parsing unit (MessageParsing Unit, MPU) is to receive and process data messages, and analyze and decode the received raw data according to a specific protocol format. Its specific responsibilities include: frame delimitation, verification, splitting and reorganization, etc. The flow association processing unit is a component specially designed to understand and analyze the connectivity and patterns in network traffic. This unit tracks each connection in the TCP / IP protocol stack, understands how each packet belongs to a specific session or stream, and extracts key information from it, such as source IP, destination IP, port number, and application layer content. The flow association processing unit also performs a certain degree of data association, but it does not associate these data packets themselves, but associates the content extracted from each message in a stream. In many cases, the content extracted from the previous message can help skip the unnecessary identification of the current and subsequent frames, thereby saving a lot of performance resources.
[0042] Step S206, determining the metadata field with the filtering rule in the packet header protocol stack of the valid message from the characteristic data, and splicing the metadata field.
[0043] In the above step S206, in network communication, each data packet contains metadata fields about its source and destination, as well as information such as the transmission protocol. These metadata fields may include information such as the source IP address, the destination IP address, the source port number, the destination port number, the protocol type, etc. The network administrator can determine the packet header protocol stack of the valid message based on these metadata fields, and formulate corresponding filtering rules to filter and process the data packets. Splicing metadata fields refers to combining these metadata fields in a certain format for easier analysis and processing. For example, the source IP address and the destination IP address can be spliced together and separated by a slash; the source port number and the destination port number can be spliced together and separated by a colon, etc. By splicing metadata fields, a unified data format can be formed to facilitate subsequent data processing and analysis.
[0044] In the embodiment of the present application, the above step S206 can be implemented by a protocol metadata extraction unit. The protocol metadata extraction unit is a core component of the deep packet inspection system. Its main task is to extract important information about the communication protocol from the network data packet. In addition, the metadata of the protocol needs to be spliced in the session to form a metadata pattern string for subsequent string matching. The splicing method of the protocol metadata follows three principles:
[0045] 1. Only the protocol metadata fields covered by the system rule collection are spliced. The protocol metadata field values that do not appear in the rules will not be spliced into the pattern string;
[0046] 2. The concatenation method of metadata pattern strings can be, for example, concatenated in the manner of "&&metadata field==metadata field value". If the rule contains a size comparison and a multi-valued integer metadata field, only the metadata field is concatenated, not the field value;
[0047] 3. The metadata pattern string is concatenated in the order of protocol parsing. Due to the concatenation method, all metadata fields are in parallel in the pattern string, so the concatenation order will not affect the matching result.
[0048] In the above principles, the first principle ensures that only valid fields included in the rules will be spliced, and fields that do not appear in the rules will not be hit by any rule. Many meaningless copying and scanning processes are omitted. In the second principle, there is no equivalence relationship between the comparison of size relationships and multi-valued integer data, and it is impossible to directly determine whether the rule is hit through pattern string matching. Only the metadata field is spliced in the pattern string. After the other sub-elements of the rule are hit, the size relationship of the integer metadata is compared. If other sub-elements do not hit, there is no need to compare the integer metadata field. The main function of the metadata pattern string is to identify the metadata information of the message in the session and match it with the metadata attributes in the preset rules. This allows us to add many metadata screening conditions in the identification rules, greatly expanding the scope of DPI filtering rules.
[0049] Step S208, performing rule matching on the spliced metadata field and the preset field to obtain a matching result, wherein the matching result includes a search rule attribute corresponding to the valid message.
[0050] In the above step S208, the concatenated metadata field and the preset field are matched by rules, which means that the metadata field in the network data packet is compared with the pre-set rules to determine whether the data packet meets the requirements of the rules. In the matching process, the data packet can be compared with the rules according to the definition and conditions of the rules to obtain a matching result. The matching result includes the retrieval rule attributes corresponding to the valid message, that is, the characteristics or attributes of the data packet that meets the rules.
[0051] In an embodiment of the present application, the above step S208 can be implemented, for example, by a string deep matching unit (hereinafter referred to as the matching unit) and an identification rule verification unit (hereinafter referred to as the verification unit). The matching unit is a module specifically used to find and analyze specific strings or pattern strings in a data packet. The matching unit includes the following functions: pattern construction, data packet traversal, metadata splicing string traversal, deep matching algorithm, complexity processing, result processing and other functions. The verification unit is a component used to verify whether the content of the data packet complies with a preset series of rules or policies. Different from the above matching unit, the verification unit only integrates and verifies the matching results of the above units, because the filtering conditions supported by traditional DPI rules are often single, or based on a single packet, and do not support information combinations between multiple data packets. Therefore, an identification rule verification unit is added to connect the matching results of different data packets in the session in series, and perform comprehensive verification to give the final recognition result. Compared with processing messages and pattern string matching, the inspection process does not consume much performance, but it can greatly expand the functionality of DPI rules. It can arbitrarily combine traditional five-tuple rules, load rules, regular filtering conditions, metadata fields, and even dynamic screening rules converted from content extracted from data packets to meet the growing DPI filtering needs.
[0052] The attributes that define the rules include "protocol number: quadruple: flow feature element: protocol metadata: regular element: load feature element, and the processing action after hitting the target". The matching and verification of the rules also follow the order of "protocol number->quadruple->protocol metadata->flow feature element->regular element->load feature element". This also complies with the matching cost of each element attribute, and the rule element with the lowest cost is matched and checked first. If the preceding element does not hit, there is no need to match the subsequent rule elements, which can save a lot of performance cost. Among them, the protocol number, quadruple and flow feature elements are relatively common, and the protocol metadata is further explained. Protocol metadata can be limited to any field of any protocol, and protocol metadata can be matched across packets in a flow. Although DPI processing does not associate messages in a flow at the flow level, protocol parsing of each packet in the flow can summarize the protocol metadata of a single packet in the flow. For example, if an HTTP uplink request and a downlink response appear in a flow, HTTP requests and HTTP responses can be limited in one rule. Even if the request and response appear in different uplink and downlink packets, accurate filtering can be performed. The HTTP example is intended to illustrate that the rule has the ability to match metadata across packets under the premise of DPI.
[0053] In the embodiment of the present application, the actual DPI system may contain millions of rules. The following uses two rules as examples to illustrate the attributes of the rules and the management method of the corresponding rule elements:
[0054] Rule 1: "TCP, *:80<->*:*, flow.pktNum==43, http.url=="helloworld", http.host="fuwuqiyuming", target list". This is a target list filtering rule that filters the number of TCP flow packets on port 80 equal to 43 and limits the HTTP protocol field.
[0055] Rule 2: "TCP, *:443<->*:*, flow.pktNum>=40, ssl.sni=="fuwuqiyuming", target list". This is a target list filtering rule that filters the number of TCP flow packets on port 443 that is greater than 40 and limits the ssl protocol field.
[0056] Combining these two rules, the protocol number rule is integrated into one protocol number rule, which is TCP. This means that the preamble will only perform rule-related processing on TCP data, and non-TCP data will be ignored. The five-tuple element only contains bidirectional port rules. The two rules are integrated into two five-tuple rules, namely the 80-port rule and the 443-port rule. All non-target ports will be ignored. The flow feature rule that limits the number of packets appears in both rules, and this field may contain a comparison of size relationships. Therefore, the rule will be integrated into flow.pktNum>=40. All flows with less than 40 packets will not be matched with subsequent protocol metadata, flow features, regular elements, and payload feature elements. This integration is intended to filter flows with less than 40 packets, but it does not change the packet number matching rule of rule one. The rule will also filter flows with 43 packets in flows with more than 40 packets. The protocol metadata contains the compound rules of HTTP host and url. This rule only targets the TCP flow of port 80 and filters the request frames that may contain HostUrl uplink data packets. The same is true for rule 2. By defining such rule attributes, the filtering attributes with strong screening force and low performance overhead can play a role first, thereby avoiding some meaningless metadata, flow characteristics, regular elements and load characteristics screening, which are also the main performance overhead in DPI engineering.
[0057] In addition, in actual DPI projects, data can hit multiple rules, but in fact the granularity between rules is not equal. When integrating the results after hitting the rules, priority should be given to outputting results with more accurate preset conditions and finer recognition levels. This relies on presetting the definition of rule priority in the inspection unit. When hitting multiple filtering rules, priority should be given to outputting recognition results with higher priority. Only when there is no conflict between the recognition results and the priorities are the same, parallel output of recognition results will be considered.
[0058] Thanks to the design of the identification rule verification unit, a multi-level rule search method can also be provided to improve the lack of flexibility in DPI rule query, that is, any unit in the rule combination can be used to perform real-time retrieval of the complete rules in the DPI system.
[0059] In an embodiment of the present application, the main storage structure of the identification rule verification unit can be a multi-level rule unit storage hash table, where the first-level rules are stored in a rule array, and each first-level rule unit contains all the elements of this rule: rule id, elements of all second-level rules, and indexes for searching each second-level rule. Each subordinate second-level rule unit is stored in a different second-level storage unit according to its rule type, for example, the five-tuple rule is stored in different five-tuple hashes according to the corresponding hash. Since any type of five-tuple filtering method is supported, there will actually be multiple five-tuple storage arrays of different types according to the attributes of the actual rules, such as a rule storage table indexed by the source direction IP, a rule storage table indexed by the source direction IP mask, and so on. Although the retrieval primary keys of these table items are different, their search principles are similar to the storage structure, so you only need to understand any one of them, and the same applies to other table items. In the actual matching process, the object hit by the rule element is the secondary rule unit. In order to be able to reversely search for its parent rule unit when the secondary rule is hit, it is necessary to record the mapping of the secondary unit to the primary rule unit. Since different primary rule units may contain the same secondary rule unit, the mapping of a secondary rule unit to a primary rule unit is one-to-many, which is stored in the secondary rule unit in the form of a primary rule id array as a search index. As long as a secondary rule unit is hit, all possible first-level rule units that may be hit can be found through the id index, which greatly improves the overall matching performance.
[0060] Through the above steps S202 to S208, the purpose of refined message detection is achieved, thereby achieving the technical effect of providing high message recognition accuracy, and further solving the technical problems of rough rule matching for message detection in related technologies, insufficient recognition accuracy and poor flexibility.
[0061] In step S202 of the above-mentioned message rule matching method, after obtaining the valid message at the network exit, the method also includes: obtaining five-tuple information of the valid message, wherein the five-tuple information includes source IP, destination IP, source address, destination address and transport layer protocol; determining the hash value corresponding to the valid message based on the five-tuple information; and determining the output path of the valid message based on the hash value.
[0062] In the embodiment of the present application, the five-tuple information refers to the source IP address, destination IP address, source port number, destination port number and transport layer protocol in the network data packet. By extracting these five-tuple information, a data packet can be uniquely identified and important information such as the source and destination addresses, protocol type, etc. of its communication can be determined. The hash value is a unique identifier calculated based on the content of the data packet. By performing hash calculation on the five-tuple information of the valid message, a unique hash value can be generated to represent the data packet. The hash value can be used to determine the routing path of the data packet, that is, to determine which node or output path in the network the valid message should be sent to. According to the pre-defined hash algorithm and routing rules, the next transmission path of the valid message can be determined according to the hash value. Through the above steps, the data traffic in the network can be more effectively managed and controlled, the transmission path of the data packet can be determined according to the five-tuple information and the hash value, and the accuracy and security of network communication can be guaranteed. These methods help to improve network performance and security and ensure that valid messages can be transmitted and processed along the correct path.
[0063] In step S204 of the above-mentioned message rule matching method, before extracting the characteristic data of the valid message, the method also includes: judging whether there is a valid message in the flow table record based on the five-tuple information of the valid message, and obtaining a first judgment result; when the first judgment result indicates that there is no valid message in the flow table record, creating a new record for storing the valid message, and assigning an UNKNOWN label to the newly created flow table record corresponding to the valid message; when the first judgment result indicates that there is a message record corresponding to the valid message in the flow table record, judging whether the label of the message record is UNKNOWN, and obtaining a second judgment result; when the second judgment result is no, no longer identifying the valid message; when the second judgment result is yes or a new record is created to store the valid message, identifying the header protocol stack and load data of the valid message; determining the five-tuple information, header protocol stack and load data of the valid message as the characteristic data of the valid message.
[0064] In step S206 of the above-mentioned message rule matching method, the metadata field having the filtering rule in the packet header protocol stack of the valid message is determined from the characteristic data, including: determining the protocol type of the valid message from the packet header protocol stack; determining whether the transport layer where each protocol type is located carries payload data; and in the case where the transport layer where each protocol type is located carries payload data, parsing the corresponding protocol to determine the metadata field.
[0065] In an embodiment of the present application, the protocol type of the valid message is determined according to the information in the characteristic data, such as TCP, UDP, IP, etc. According to the protocol type, it is determined whether the protocol carries load data in the transport layer. For example, TCP and UDP protocols usually carry load data, while IP protocol does not carry load data. For protocol types that carry load data, such as TCP and UDP, it is necessary to further parse the protocol to determine the metadata fields therein, such as source port number, destination port number, sequence number, confirmation number and other information. These metadata fields can be used for further analysis and processing of valid messages. Through the above steps, the metadata fields with filtering rules in the packet header protocol stack of the valid message can be accurately determined from the characteristic data, including the protocol type, load data situation, and the corresponding protocol is parsed to obtain more detailed metadata field information.
[0066] In step S208 of the message rule matching method, the spliced metadata field and the preset field are matched with the rules to obtain a matching result, including: obtaining a valid payload in a valid message; obtaining an initial matching result based on the payload, the spliced metadata field and the preset field, wherein the initial matching result is used to represent the matching result of the secondary rule unit; determining the matching result based on the initial matching result, wherein the matching result is used to represent the matching result of the primary rule unit, and the primary rule unit is the parent rule unit of the secondary rule unit.
[0067] In an embodiment of the present application, a payload, that is, the data content carried by the payload, is extracted from a valid message. The payload generally refers to the actual data transmitted in the data packet, rather than the header information. The payload is matched with the spliced metadata field and the preset field to obtain an initial matching result. This initial matching result is used to indicate the matching status of the secondary rule unit, that is, the specific application of the rule is determined based on the matching status of the payload and the metadata field. Based on the initial matching result, the matching result is further determined, wherein the matching result is used to indicate the matching status of the primary rule unit. The primary rule unit refers to the parent rule unit of the secondary rule unit, that is, the matching status of the overall rule is determined based on the initial matching result.
[0068] In the above steps, an initial matching result is obtained based on the payload, the concatenated metadata field and the preset field, including: determining the payload as the first search mother string, and determining the concatenated metadata field as the second search mother string; determining the first search substring for payload matching from the preset field, and determining the second search substring for metadata matching; matching the first search mother string and the first search substring, and matching the second search mother string and the second search substring to obtain an initial matching result.
[0069] In an embodiment of the present application, the payload is used as the first search mother string, that is, the main data that needs to be matched. At the same time, the spliced metadata field is used as the second search mother string, that is, the metadata information that needs to be matched with the payload. A first search substring for matching the payload is determined in the preset field, which is a keyword or pattern for matching the payload. At the same time, a second search substring for matching the metadata field is determined, which is a keyword or pattern for matching the metadata field. The initial matching results can be obtained by matching the first search mother string and the first search substring, and matching the second search mother string and the second search substring. These results will reflect the matching of the payload and the metadata field in the preset field, thereby determining the application of the rules and the matching results.
[0070] In the above steps, the matching result is determined based on the initial matching result, including: determining the id number of the secondary rule unit from the initial matching result; determining all parent rule units corresponding to the id number of the secondary rule unit from the multi-level rule unit storage table; and determining the parent rule unit as the matching result.
[0071] In an embodiment of the present application, the ID number of the secondary rule unit, that is, the unique identifier of the rule unit, is determined according to the initial matching result. According to the ID number of the secondary rule unit, all parent rule units corresponding to the ID number are searched in the multi-level rule unit storage table. These parent rule units include primary rule units, higher-level rule units of secondary rule units, and other related rule units. All parent rule units determined from the multi-level rule unit storage table are used as the final matching results. These parent rule units will summarize the matching results for the payload and metadata fields, as well as the application of the rules. The matching result is determined according to the initial matching result, including determining the corresponding rule unit ID number, searching for all related parent rule units, and using it as the final matching result. This hierarchical rule matching and result determination method helps to have a more comprehensive understanding of the matching of the rules, while making rule management and application more flexible and efficient.
[0072] In order to better understand the above message rule matching method provided in the embodiment of the present application, the above steps are explained below:
[0073] Step 1: The data acquisition unit first uses the flow aggregation equipment deployed in the exit area or core area of the network flow to aggregate the flow in multiple optical fibers, filter out the retransmitted data packets caused by network delays, error packets of failed requests, and other non-human generated data packets, and then represents the flow hash corresponding to the i-th frame message according to the hash value of the remaining data packets hashi=CRC(srcipi, srcporti, dstipi, dstporti, protoi). Finally, according to the hash value and the number of parallel message processing units, all data packets are forwarded to the I / O interfaces of multiple parallel message processing units through load balancing technology.
[0074] Step 2: The message processing unit determines whether there is a corresponding record in the flow table ST = {(srcip1, srcport1, dstip1, dstport1, proto1): label1, .., (srcipτ, srcportτ, dstipτ, dstportτ, protoτ): labelτ} according to the five-tuple of the data message. If it exists and labeli is not UNKNOWN, the flow label is directly assigned, the identification result in the flow is inherited, and the current message is no longer processed. If the corresponding record in the flow table does not exist, a new flow table node is created and an UNKNOWN flow label is assigned to it for subsequent processing. If the corresponding record in the flow table already exists, the flow label is UNKNOWN and the processing continues. After frame delimitation, verification, splitting and reassembly, the five-tuple information, application layer load, and packet protocol stack extracted from the message are stored in the flow record. The original data source for the subsequent identification rule verification unit to match and verify the rules, the five-tuple rule matching in the subsequent identification rules, and the protocol rule matching is the corresponding flow record in the flow table in the message processing unit here. Through the flow table record, the association relationship between messages is constructed.
[0075] Step 3: The protocol metadata extraction unit extracts the necessary high-level protocol data based on the packet header protocol stack obtained by the message processing unit, such as "Ethernet: IP: UDP: GPRS: IP: TCP: HTTP", and the starting position of the header of each layer of the protocol. At the same time, the protocol metadata extraction unit will process according to the type of transport layer, the presence or absence of data load and other conditions, and use the transport layer payload of some messages as the input of the subsequent string deep matching unit. Finally, the protocol metadata obtained by the unit is effectively spliced, for example, #http.host=="www.texaslotto.com"&&http.request.method=="GET".
[0076] Figure 3 is a processing flow chart of a protocol metadata extraction unit according to an embodiment of the present application, such as Figure 3 As shown, the protocol metadata extraction unit performs classification processing according to the transport layer protocol. If it is TCP stream high-level protocol processing, the high-level protocol is identified based on the prediction of the flow information, and it is determined whether TCP carries a load. If there is no load, it is not parsed and the process ends. If it carries a load, the high-level protocol is polled to identify, and the corresponding high-level protocol is interpreted, and the high-level protocol metadata fields are spliced. If it is UDP stream high-level protocol processing, it is necessary to determine whether UDP carries a load. If there is no load, it is not parsed and the process ends. If it carries a load, the high-level protocol is polled to identify, and the corresponding high-level protocol is interpreted, and the high-level protocol metadata fields are spliced. If it is other protocol high-level protocol processing, the high-level protocol is identified based on the prediction of the flow information, the high-level protocol is polled to identify, and the corresponding high-level protocol is interpreted, and the high-level protocol metadata fields are spliced.
[0077] Step 4: The string deep matching unit is responsible for using the payload in the data packet and the spliced protocol metadata string as the search mother string, and using pre-set keywords, strings, wildcards, and regular expressions as search substrings for efficient identification and matching. In order to achieve such functions, specially designed libraries or tools may be used, such as libpcap (for capturing network traffic), pypcap, or scapy (Python library for packet parsing and construction). The specific code implementation depends on the language and technology stack used, but the core logic usually includes reading the data packet, decoding the application layer data, and then performing a string matching operation. In actual DPI systems, this matching is often highly optimized because a large amount of real-time network traffic needs to be processed. Therefore, efficient algorithms and data structures (such as suffix arrays, Trie trees, or Aho-Corasick automata) are the first choice for implementation, and since the design of this application is not limited to specific detailed implementations, the intermediate matching implementation process will not be described in detail. After payload matching and metadata matching, the deep matching unit will return the corresponding matching result. The result is presented in the form of the id number of each secondary rule unit, such as "id payload = 1, 16, id protocol = 1, 8", and output to the identification rule verification unit.
[0078] Step 5: The identification rule verification unit will search for all possible first-level rule units that may be hit according to the hit status of the second-level rule unit. If there is any second-level rule unit that is not hit under a first-level rule, there is no need to examine other second-level rule units, and the first-level rule unit can be judged as not hit, thereby eliminating some unnecessary rule retrieval processes. For example, the attribute of a rule is #http.host=="www.texaslotto.com"&&http.request.method=="POST", and the rule attribute parsed in the actual message is #http.host=="www.texaslotto.com"&&http.request.method=="GET". When it is found that the http.request.method second-level field does not match during the comparison process, all other second-level rule attributes under the first-level rule do not need to be compared and recorded. After all the hit second-level rule units are screened through the above process, all the first-level rule units that are actually hit are obtained.
[0079] Figure 4 is a structural diagram of a message rule matching device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:
[0080] An acquisition module 40 is used to acquire valid messages at a network exit;
[0081] An extraction module 42, used to extract characteristic data of valid messages;
[0082] A determination module 44, used to determine the metadata field with the filtering rule in the packet header protocol stack of the valid message from the characteristic data, and splice the metadata field;
[0083] The matching module 46 is used to perform rule matching on the spliced metadata field and the preset field to obtain a matching result, wherein the matching result includes a search rule attribute corresponding to the valid message.
[0084] In the acquisition module of the matching device of the above-mentioned message rules, the acquisition module is also used to obtain five-tuple information of the valid message, wherein the five-tuple information includes source IP, destination IP, source address, destination address and transport layer protocol; determine the hash value corresponding to the valid message based on the five-tuple information; and determine the output path of the valid message based on the hash value.
[0085] In the extraction module of the matching device of the above-mentioned message rules, the extraction module is also used to judge whether there is a valid message in the flow table record based on the five-tuple information of the valid message, and obtain a first judgment result; when the first judgment result indicates that there is no valid message in the flow table record, a new record is created to store the valid message, and an UNKNOWN label is assigned to the new flow table record corresponding to the valid message; when the first judgment result indicates that there is a message record corresponding to the valid message in the flow table record, it is judged whether the label of the message record is UNKNOWN to obtain a second judgment result; when the second judgment result is no, the valid message is no longer identified; when the second judgment result is yes or a new record is created to store the valid message, the header protocol stack and load data of the valid message are identified; the five-tuple information, header protocol stack and load data of the valid message are determined as the characteristic data of the valid message.
[0086] In the determination module of the matching device of the above-mentioned message rules, the determination module is also used to determine the protocol type of the valid message from the packet header protocol stack; determine whether the transport layer where each protocol type is located carries payload data; and when the transport layer where each protocol type is located carries payload data, parse the corresponding protocol to determine the metadata field.
[0087] In the matching module of the above-mentioned message rule matching device, the matching module is also used to obtain the payload in the valid message; based on the payload, the spliced metadata field and the preset field, an initial matching result is obtained, wherein the initial matching result is used to represent the matching result of the secondary rule unit; based on the initial matching result, a matching result is determined, wherein the matching result is used to represent the matching result of the primary rule unit, and the primary rule unit is the parent rule unit of the secondary rule unit.
[0088] In the matching module of the matching device of the above-mentioned message rules, the matching module is also used to determine the payload as the first search mother string, and determine the spliced metadata field as the second search mother string; determine the first search substring for payload matching from the preset field, and determine the second search substring for metadata matching; match the first search mother string and the first search substring, and match the second search mother string and the second search substring to obtain an initial matching result.
[0089] In the matching module of the above-mentioned message rule matching device, the matching module is also used to determine the id number of the secondary rule unit from the initial matching result; determine all parent rule units corresponding to the id number of the secondary rule unit from the multi-level rule unit storage table; and determine the parent rule unit as the matching result.
[0090] It should be noted that Figure 4 The message rule matching device shown is used to perform Figure 2The message rule matching method shown in the figure, therefore the relevant explanations in the above message rule matching method are also applicable to the message rule matching device, which will not be repeated here.
[0091] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute program instructions to implement the following functions: obtain a valid message at a network exit; extract feature data of the valid message; determine a metadata field with a filtering rule in a packet header protocol stack of the valid message from the feature data, and splice the metadata field; perform rule matching on the spliced metadata field and a preset field to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
[0092] It should be noted that the above electronic equipment is used to execute Figure 2 The message rule matching method shown in the figure, therefore the relevant explanations in the above message rule matching method are also applicable to the electronic device and will not be repeated here.
[0093] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following message rule matching method by running the computer program: obtaining a valid message at a network exit; extracting feature data of the valid message; determining a metadata field having a filtering rule in a packet header protocol stack of the valid message from the feature data, and splicing the metadata field; performing rule matching on the spliced metadata field and a preset field to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
[0094] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The message rule matching method shown in the figure, therefore the relevant explanations in the above message rule matching method are also applicable to the non-volatile storage medium, and will not be repeated here.
[0095] An embodiment of the present application further provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the message rule matching method in each embodiment of the present application.
[0096] The embodiments of the present application also provide a computer program, which, when executed by a processor, implements the steps of the message rule matching method in each embodiment of the present application.
[0097] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0098] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0103] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A message rule matching method, characterized in that: include: Obtain valid messages at the network exit; Extracting characteristic data of the valid message; Determine the metadata field with the filtering rule in the packet header protocol stack of the valid message from the characteristic data, and splice the metadata field; The spliced metadata field and the preset field are matched with rules to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
2. The method according to claim 1, characterized in that: After obtaining the valid message at the network egress, the method further includes: Acquire the five-tuple information of the valid message, wherein the five-tuple information includes the source IP, destination IP, source address, destination address and transport layer protocol; Determine the hash value corresponding to the valid message according to the quintuple information; An output path of the valid message is determined according to the Hash value.
3. The method according to claim 2, characterized in that Before extracting the characteristic data of the valid message, the method further includes: According to the five-tuple information of the valid message, determine whether the valid message exists in the flow table record to obtain a first determination result; If the first judgment result indicates that the valid message does not exist in the flow table record, create a new record for storing the valid message, and assign an UNKNOWN label to the new flow table record corresponding to the valid message; In the case where the first judgment result indicates that there is a message record corresponding to the valid message in the flow table record, judging whether the label of the message record is UNKNOWN to obtain a second judgment result; If the second judgment result is no, no longer identifying the valid message; When the second judgment result is yes or a new record is created to store the valid message, the header protocol stack and payload data of the valid message are identified; and the quintuple information of the valid message, the header protocol stack and the payload data are determined as characteristic data of the valid message.
4. The method according to claim 1, characterized in that Determining the metadata field having the filtering rule in the packet header protocol stack of the valid message from the characteristic data includes: Determine the protocol type of the valid message from the packet header protocol stack; Determine whether the transport layer of each protocol type carries payload data; In the case where the payload data is carried in the transport layer where each protocol type is located, the corresponding protocol is parsed to determine the metadata field.
5. The method according to claim 1, characterized in that: The spliced metadata fields are matched with the preset fields according to the rules to obtain the matching results, including: Obtaining a payload in the valid message; Obtaining an initial matching result according to the payload, the concatenated metadata field and the preset field, wherein the initial matching result is used to represent a matching result of a secondary rule unit; The matching result is determined according to the initial matching result, wherein the matching result is used to represent the matching result of the first-level rule unit, and the first-level rule unit is the parent rule unit of the second-level rule unit.
6. The method according to claim 5, characterized in that According to the payload, the concatenated metadata field and the preset field, an initial matching result is obtained, including: Determine the payload as a first search mother string, and determine the concatenated metadata field as a second search mother string; Determining a first search substring for performing payload matching from the preset field, and determining a second search substring for performing metadata matching; The first search mother string and the first search substring are matched, and the second search mother string and the second search substring are matched to obtain the initial matching result.
7. The method according to claim 5, characterized in that Determining the matching result according to the initial matching result includes: Determine the ID number of the secondary rule unit from the initial matching result; Determine all parent rule units corresponding to the ID number of the secondary rule unit from the multi-level rule unit storage table; The parent rule unit is determined as the matching result.
8. A message rule matching device, characterized in that: include: An acquisition module is used to acquire valid messages at the network exit; An extraction module, used to extract characteristic data of the valid message; A determination module, used to determine the metadata field with the filtering rule in the packet header protocol stack of the valid message from the characteristic data, and splice the metadata field; The matching module is used to perform rule matching on the spliced metadata field and the preset field to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor connected to the memory and used to execute program instructions to implement the following functions: obtaining a valid message at a network exit; extracting feature data of the valid message; determining a metadata field having a filtering rule in a packet header protocol stack of the valid message from the feature data, and splicing the metadata field; The spliced metadata field and the preset field are matched with rules to obtain a matching result, wherein the matching result includes a retrieval rule attribute corresponding to the valid message.
10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the message rule matching method described in any one of claims 1 to 7 by running the computer program.
11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the message rule matching method described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Network traffic analysis method and device and storage medium
CN120263542A
Network traffic analysis method, device and storage medium
CN120263542B
ACL rule processing method and device, product, equipment and medium
CN121396665A
