A complex event processing method and system based on network traffic metadata

Abstract

The present invention provides a complex event processing method and system based on network traffic metadata. The method includes collecting network traffic metadata from a switch mirror port and saving it as a pcap package, calling a parsing instruction to parse the detailed information in the pcap package to obtain the parsing results; using a rule designer with a visual / graphical interface and pre-designing complex event processing rules based on an operator model, using an AI model for testing and outputting rule text; using a rete tree structure to store the rule text to obtain a rete rule tree, which is then saved to a rule database based on permissions; starting from the root node of the rete rule tree, matching the pattern corresponding to each type node with the facts / business data generated by the metadata, and outputting the results that match the pattern corresponding to the type node to external storage. This method implements manually predefined business rules and integrates machine learning and artificial intelligence models, enabling multi-faceted detection of known security threats and perception of unknown network threats.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a complex event processing method and system based on network traffic metadata. Background Art

[0002] Unknown vulnerabilities are prevalent across operating systems, application software, network devices, security equipment, and business systems. This poses even more severe challenges to network security amidst the growing commercialization of cyber weapons and the proliferation of organized cyberattacks. Traditional security monitoring methods, largely based on a known rule base, can detect known security threats but are powerless against unknown ones. Furthermore, they are unable to fully trace the source of ongoing or already-inflicted intrusions and assess the damage.

[0003] Based on the above points, this paper proposes a complex event engine (Sandbox) based on Flink and Esper. This engine can implement predefined business rules by business experts, integrate and execute machine learning and artificial intelligence models, and use real-time streaming computing to build diversified operator capabilities, realizing multi-faceted and multi-angle detection of known security threats and awareness of unknown network threats.

[0004] During the initial research and planning process for using Flink and Esper as technical solutions, this paper discovered a series of pain points. The specific pain points are as follows:

[0005] 1. It cannot perform semantic optimization and is not convenient for dynamic rule updates. Security analysis scenarios have extremely flexible requirements, and other CEP engine-implemented products will encounter many problems when requirements change drastically.

[0006] 2. Insufficient high availability support. Products implemented by other CEP engines lack some essential operators, such as the "no-occurrence operator." In a common application scenario, a rule specifies that if no system logs are received from a server for a long period of time, the server is considered to have experienced an anomaly and the user must be notified promptly. Furthermore, products implemented by other CEP engines lack the ability to customize operators based on specific business types.

[0007] 3. Insufficient integration of machine learning and artificial intelligence. Few products implemented by other CEP engines integrate machine learning and artificial intelligence models. Summary of the Invention

[0008] The present invention proposes a complex event processing method and system based on network traffic metadata to address the above-mentioned defects of the prior art.

[0009] In one aspect, the present invention provides a complex event processing method based on network traffic metadata, the method comprising the following steps:

[0010] S1: Collect the network traffic metadata of the switch mirror port and save it as a pcap package. Call the parsing instruction to parse the detailed information of the pcap package to obtain the parsing result, and then save the parsing result to the Kafka message queue as the data source;

[0011] S2: Pre-design complex event processing rules based on operator models using a rule designer with a visual / graphical interface. The complex event processing rules and corresponding processing resources are allocated according to permissions. The allocation results are tested using an AI model, and the test results are saved as rule text.

[0012] S3: Decomposing the processing rules of the complex events in the rule text into a number of minimum rule matching items, taking a minimum rule matching item as a pattern, and then taking a pattern as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree, and then saving the rete rule tree into the rule database according to the authority;

[0013] S4: Obtain the data source from the Kafka message queue and store the data source as facts / business data in the memory. Starting from the root node of the rete rule tree, match the pattern corresponding to each type node with the facts / business data. If the matching results are the same, output the results that match the pattern corresponding to the type node to external storage.

[0014] The above method has launched a new CEP engine based on the CEP engines of Flink and Esper. It uses network traffic metadata to trace back and analyze data packet characteristics and abnormal network behavior, detect network attacks in real time, and discover advanced unknown attacks that have been lurking for a long time. This engine consists of three parts: data source (Source), complex event engine (Sandbox), and data sink (Sink). The data source (Source) comes from the Kafka traffic metadata Topic, the complex event engine (Sandbox) consists of dynamic multi-level rule flow, custom operators, streaming statistics and machine learning, and the data sink (Sink) is to the Kafka event Topic. The present invention can not only realize the pre-definition of business rules by business experts, but also integrate the execution of machine learning and artificial intelligence models and use real-time streaming computing to build diversified operator capabilities, realizing the detection of known security threats and the perception of unknown network threats from multiple angles.

[0015] In a specific embodiment, the step of collecting data from the switch mirror port and saving it as a pcap packet includes:

[0016] Start the main service listening port configuration, then query the mirror port configuration from the database, configure the packet capture program parameters on each mirror port and collect data.

[0017] In a specific embodiment, the calling of the parsing instruction to parse the detailed information of the pcap package to obtain a parsing result, and then saving the parsing result to the Kafka message queue as a data source, specifically includes the following steps:

[0018] Monitor the Kafka message queue to determine whether the pcap package is generated;

[0019] If so, obtain the file name of the pcap package, call the parsing instruction to parse the network traffic metadata in the pcap package and write the parsing result to the Kafka metadata Topic;

[0020] If not, continue to monitor the Kafka message queue.

[0021] In a specific embodiment, the operator model specifically includes: built-in operators and user-defined operators.

[0022] In a specific embodiment, the rule designer with a visual / graphical interface includes:

[0023] A visual / graphical interface for defining the processing rules by mouse clicks;

[0024] A visual / graphical interface for combining multiple conditions for the processing rules and displaying them graphically. Based on the above method, complex business processing rules can be defined with just a click of the mouse, and the multiple conditional combinations of the rules are also displayed graphically. This allows even ordinary business personnel without any programming experience to easily get started and complete the definition of complex business processing rules. Because all business processing rule designers are web-based and rule definitions are completed with a click of the mouse, the entry threshold for ordinary users to use various designers is extremely low, making it easy to define the desired business processing rules based on business needs.

[0025] In a specific embodiment, the test includes: a quick test, a simulation test, and a Rest service test.

[0026] In a specific embodiment, the processing rules include wizard rules and script rules.

[0027] In a specific embodiment, the pattern is used as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree. The specific steps include:

[0028] Step 1: Create a root node of a rete tree structure;

[0029] Step 2: Take a pattern from the processing rule and record it as pattern i, where i is the pattern number and i = {1, 2, 3...}, check the parameter type in pattern i, and if the parameter type is a new fact type, add a type node to the rete tree structure;

[0030] Step 3: record the Alpha node corresponding to the pattern i as node Alpha(i), check whether the node Alpha(i) already exists, if so, record the position of the node Alpha(i), if not, add the pattern i as a new Alpha node to the rete tree structure, and then establish the Alpha memory table of the corresponding node Alpha(i) according to the pattern i;

[0031] Step 4: Repeat steps 2 to 3 and increment i by 1 after each repetition until all patterns are processed;

[0032] Step 5: Assemble the Beta nodes of the rete tree structure, specifically including the following steps:

[0033] According to the mode i, the Beta node of the mode i is recorded as node Beta(i);

[0034] When i=2, let the left input node of node Beta(2) be node Alpha(1) and the right input node be node Alpha(2);

[0035] When i>2, the left input node of node Beta(i) is made into node Beta(i-1) and the right input node is made into node Alpha(i), and the memory tables of the two parent nodes of the node Beta(i) are inlined into the memory table of the node Beta(i);

[0036] Step 6: Repeat step 5 until all Beta nodes are processed;

[0037] Step 7: Encapsulate the action into a leaf node as the output node of the node Beta(i);

[0038] Step 8: Use the final obtained rete tree structure as the rete rule tree.

[0039] In a specific embodiment, in S4, if the matching results are not the same, the following steps are performed:

[0040] Step a: passing the fact / business data to the Alpha node, detecting whether the fact / business data matches the pattern corresponding to the current Alpha node, and if so, outputting the result that matches the pattern corresponding to the current Alpha node to external storage;

[0041] If not, pass the fact / business data to the next Alpha node and repeat this step. If there is no match, execute steps b and c.

[0042] Step b: Pass the fact / business data to the left end of the Beta node, then encapsulate the fact / business data into a list containing only one fact / business data, and use the list as a token;

[0043] Step c: Pass the fact / business data to the right end of the Beta node and the Token in the left storage area for matching;

[0044] If the match is successful, the result that matches the pattern corresponding to the current Beta node is output to the external storage;

[0045] If the match is unsuccessful, the fact / business data is added to the token, and the token is passed to the next node and this step is repeated;

[0046] Step d: When the Token is passed to the final node, the result that meets the pattern corresponding to the final node is output to the external storage.

[0047] In a specific embodiment, the corresponding processing resources include all requested URLs.

[0048] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer processor, the above method is implemented.

[0049] According to a third aspect of the present invention, a complex event processing system based on network traffic metadata is proposed, the system comprising:

[0050] Network metadata parsing module: configured to collect network traffic metadata from the switch mirror port and save it as a pcap package, call parsing instructions to parse the detailed information of the pcap package to obtain parsing results, and then save the parsing results to the Kafka message queue as a data source;

[0051] Rule design module: configured to pre-design complex event processing rules based on operator models using a rule designer with a visual / graphical interface, wherein the complex event processing rules and corresponding processing resources are allocated according to permissions, and the allocation results are tested using an AI model, and the test results are saved as rule text;

[0052] A rule tree construction module is configured to decompose the processing rules of complex events in the rule text into a number of minimum rule matching items, take a minimum rule matching item as a pattern, and then take a pattern as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree, and then saving the rete rule tree to the rule database according to the authority;

[0053] Metadata rule matching module: configured to obtain the data source from the Kafka message queue and store the data source as facts / business data in memory, starting from the root node of the rete rule tree, matching the pattern corresponding to each type node with the facts / business data, and if the matching results are the same, outputting the results that match the pattern corresponding to the type node to external storage.

[0054] The present invention collects network traffic metadata from the switch's mirrored port and saves it as a pcap package. A parsing instruction is then called to parse the detailed information of the pcap package to obtain a parsing result. A rule designer with a visual / graphical interface pre-designs complex event processing rules based on an operator model, and an AI model is used for testing and outputting rule text. A rete tree structure is used to store the rule text to obtain a rete rule tree, which is then saved to a rule database based on permissions. Starting from the root node of the rete rule tree, the pattern corresponding to each type node is matched with the facts / business data generated by the metadata, and the results that match the pattern corresponding to the type node are output to an external storage device. This method implements both manually predefined business rules and the integration of machine learning and artificial intelligence models, enabling the detection of known security threats and the perception of unknown network threats from multiple perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many of the intended advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. Other features, objects, and advantages of the present application will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0057] Figure 2 is a flow chart of a complex event processing method based on network traffic metadata according to an embodiment of the present invention;

[0058] Figure 3 This is a flow chart of network traffic packet capture according to a specific embodiment of the present invention;

[0059] Figure 4 is a flowchart of network traffic metadata parsing according to a specific embodiment of the present invention;

[0060] Figure 5 is a flowchart of a rule designer according to a specific embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of authority allocation roles in a specific embodiment of the present invention;

[0062] Figure 7 It is a visual rete tree diagram of a specific embodiment of the present invention;

[0063] Figure 8 is a flow chart of a rule execution engine according to a specific embodiment of the present invention;

[0064] Figure 9 It is a system overall architecture diagram of a specific embodiment of the present invention;

[0065] Figure 10 This is a framework diagram of a complex event processing system based on network traffic metadata according to an embodiment of the present invention;

[0066] Figure 11 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0067] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0068] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0069] Figure 1An exemplary system architecture 100 is shown to which a complex event processing method based on network traffic metadata according to an embodiment of the present application can be applied.

[0070] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0071] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications can be installed on terminal devices 101, 102, 103, such as data processing applications, data visualization applications, web browser applications, etc.

[0072] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitations are given here.

[0073] The server 105 may be a server that provides various services, such as a background information processing server that supports the network traffic metadata displayed on the terminal devices 101, 102, and 103. The background information processing server may process the obtained rule text and generate a processing result (such as a rete rule tree).

[0074] It should be noted that the method provided in the embodiment of the present application can be executed by the server 105 or by the terminal devices 101, 102, and 103. The corresponding device is generally set in the server 105 and can also be set in the terminal devices 101, 102, and 103.

[0075] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0076] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0077] According to an embodiment of the present invention, a complex event processing method based on network traffic metadata is provided. Figure 2 FIG. 1 is a flow chart showing a complex event processing method based on network traffic metadata according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0078] S1: Collect the network traffic metadata of the switch mirror port and save it as a pcap package. Call a parsing instruction to parse the detailed information of the pcap package to obtain a parsing result, and then save the parsing result to the Kafka message queue as a data source.

[0079] Figure 3 The following is a flowchart of network traffic packet capture according to a specific embodiment of the present invention. In this embodiment, the specific steps of the network traffic packet capture process include:

[0080] Start the main service listening port configuration, and then query the mirror port configuration from the database. If the location of the mirror port is queried, configure the packet capture program parameters at each mirror port and collect data. If the location of the mirror port is not queried, continue to listen to the port configuration.

[0081] Figure 4 A flowchart of network traffic metadata parsing according to a specific embodiment of the present invention is shown. The specific process is as follows:

[0082] Monitor the Kafka message queue to determine whether the pcap package is generated;

[0083] If so, obtain the file name of the pcap package, call the parsing instruction to parse the network traffic metadata in the pcap package and write the parsing result to the Kafka metadata Topic;

[0084] If not, continue to monitor the Kafka message queue;

[0085] If an exception occurs when calling the parsing instruction to parse the pcap package, the system returns to monitoring the Kafka message queue.

[0086] S2: Pre-design processing rules for complex events based on operator models through a rule designer with a visual / graphical interface, wherein the processing rules and corresponding processing resources of complex events are allocated according to permissions, and the results of the allocation are tested using an AI model, and the test results are saved as rule text.

[0087] In a specific embodiment, the corresponding processing resources include all requested URLs.

[0088] This step mainly provides a graphical interface for rule design, which is divided into two parts: design and permission control. The design part is completed by the rule designer, and the control part is completed based on permission allocation.

[0089] Figure 5 A flowchart of a rule designer for a specific embodiment of the present invention is shown. The design part mainly provides visualization tools to design flexible and diverse rules. All rule designers are visual and graphical designers. Complex business rule definitions can be achieved through mouse clicks, and the multi-condition combination of rules is also presented in a graphical manner. In this way, even ordinary business personnel without any programming experience can easily get started and complete the definition of complex business rules. Because all business rule designers are based on web pages, and the definition of rules is completed by mouse clicks, for an ordinary user, the entry threshold for using various designers is extremely low, and it is easy to define the desired business rules in combination with business needs.

[0090] Figure 6 A schematic diagram of permission allocation roles in a specific embodiment of the present invention is shown, indicating that users, rules, resources and other contents can be controlled by roles.

[0091] In a specific embodiment, the rule designer with a visual / graphical interface includes:

[0092] A visual / graphical interface for defining the processing rules by mouse clicks;

[0093] A visualization / graphical interface for combining multiple conditions of the processing rules and displaying them in a graphical manner.

[0094] In a specific embodiment, the operator model specifically includes: built-in operators and user-defined operators.

[0095] In a specific embodiment, the test includes: a quick test, a simulation test, and a Rest service test.

[0096] In a specific embodiment, the processing rules include wizard rules and script rules.

[0097] S3: Decompose the processing rules of the complex events in the rule text into several minimum rule matching items, take a minimum rule matching item as a pattern, and then take the pattern as a type node to construct a rete tree structure, so as to store the rule text in the rete tree structure to obtain a rete rule tree, and then save the rete rule tree into the rule database according to the authority.

[0098] In a specific embodiment, a model building engine is used to connect the rule designer and model execution, converting the rule text designed by the designer into a RETE tree structure that the engine can recognize. This structure is then saved to the rule database based on permissions, providing dynamic injection conditions for the model execution engine. Therefore, the implementation of RETE in this embodiment can be divided into two parts: rule compilation and runtime execution. Rule compilation refers to the process of generating an inference network based on a rule set, while runtime execution refers to the process of feeding data into the inference network for screening. Rule compilation is the process of generating a network based on inference from a rule file.

[0099] Figure 7 A visualization rete tree diagram of a specific embodiment of the present invention is shown below. Figure 7 The visual rete tree diagram shown is used to illustrate the rete rule compilation process in this embodiment:

[0100] 1. The root node (701) is the entrance for all objects to enter the network, and then enter the type node Packet1 (702) and type node Packet2 (703);

[0101] 2. Type node Packet1 (702) and type node Packet2 (703) are our facts, which are the pojo used in our rules. Each fact is a type node. Type checking is performed on the type node. The engine only allows objects that match the Object type to reach the node. It can propagate to the Alpha node (704), the left input node of the Beta node (705), and the Beta node (706).

[0102] 3. Alpha node (704) is used to evaluate literal conditions, such as the literal condition: protocol = "modbus". When a rule has multiple literal conditions, these literal conditions are linked together and another rule, operate = "remote control" (707), is added to the condition of Alpha node (704);

[0103] 4. Beta node (706) is used to compare and check two objects. It is agreed here that the two inputs of Beta node are called left side (Join Node) and right side; the left side is usually a list of objects, and the right side (NotNode) is usually a single object; each Beta node has its own terminal node and other components; Beta node has memory function; the input on the left is called Beta Memory, which will remember all the semantics that have arrived; the input on the right is called Alpha Memory, which will remember all the objects that have arrived. Therefore, the function of the left input node (705) of Beta node (706) in the figure is to convert a single Object into a single object array (single Object Tuple) and propagate it to JoinNode node; because we mentioned above that the left input is usually a list of objects;

[0104] 5. Finally, the terminal nodes (709 and 710) are reached through the nodes described above. The terminal node indicates that a single rule matches all conditions. It should be noted that there are multiple terminal nodes in the network. When a single rule contains "or", multiple terminal nodes will also be generated.

[0105] In a specific embodiment, the pattern is used as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree. The specific steps include:

[0106] Step 1: Create a root node of a rete tree structure;

[0107] Step 2: Take a pattern from the processing rule and record it as pattern i, where i is the pattern number and i = {1, 2, 3...}, check the parameter type in pattern i, and if the parameter type is a new fact type, add a type node to the rete tree structure;

[0108] Step 3: record the Alpha node corresponding to the pattern i as node Alpha(i), check whether the node Alpha(i) already exists, if so, record the position of the node Alpha(i), if not, add the pattern i as a new Alpha node to the rete tree structure, and then establish the Alpha memory table of the corresponding node Alpha(i) according to the pattern i;

[0109] Step 4: Repeat steps 2 to 3 and increment i by 1 after each repetition until all patterns are processed;

[0110] Step 5: Assemble the Beta nodes of the rete tree structure, specifically including the following steps:

[0111] According to the mode i, the Beta node of the mode i is recorded as node Beta(i);

[0112] When i=2, let the left input node of node Beta(2) be node Alpha(1) and the right input node be node Alpha(2);

[0113] When i>2, the left input node of node Beta(i) is made into node Beta(i-1) and the right input node is made into node Alpha(i), and the memory tables of the two parent nodes of the node Beta(i) are inlined into the memory table of the node Beta(i);

[0114] Step 6: Repeat step 5 until all Beta nodes are processed;

[0115] Step 7: Encapsulate the action (Then part) into a leaf node (Action node) as the output node of the node Beta(i);

[0116] Step 8: Use the final obtained rete tree structure as the rete rule tree.

[0117] In the above steps, for example, if the processing rule is: (age>10, age<20), then age>10 is one pattern, and age<20 is another pattern.

[0118] S4: Obtain the data source from the Kafka message queue and store the data source as facts / business data in the memory. Starting from the root node of the rete rule tree, match the pattern corresponding to each type node with the facts / business data. If the matching results are the same, output the results that match the pattern corresponding to the type node to external storage.

[0119] In a specific embodiment, in S4, if the matching results are not the same, the following steps are performed:

[0120] Step a: passing the fact / business data to the Alpha node, detecting whether the fact / business data matches the pattern corresponding to the current Alpha node, and if so, outputting the result that matches the pattern corresponding to the current Alpha node to external storage;

[0121] If not, pass the fact / business data to the next Alpha node and repeat this step. If there is no match, execute steps b and c.

[0122] Step b: Pass the fact / business data to the left end of the Beta node, then encapsulate the fact / business data into a list containing only one fact / business data, and use the list as a token;

[0123] Step c: Pass the fact / business data to the right end of the Beta node and the Token in the left storage area for matching;

[0124] If the match is successful, the result that matches the pattern corresponding to the current Beta node is output to the external storage;

[0125] If the match is unsuccessful, the fact / business data is added to the token, and the token is passed to the next node and this step is repeated;

[0126] Step d: When the Token is passed to the final node, the result that meets the pattern corresponding to the final node is output to the external storage.

[0127] In this embodiment, the rule execution engine is used to implement the steps in S4. Figure 8 A flow chart of a rule execution engine according to a specific embodiment of the present invention is shown.

[0128] Figure 9 FIG. 1 shows a system overall architecture diagram of a specific embodiment of the present invention, as shown in FIG. Figure 9 As shown, a detection system for network security is formed based on network traffic, visually definable rules, auxiliary rule construction models and rule operation models.

[0129] Key words in this invention:

[0130] 1. Port mirroring: To facilitate traffic analysis on one or more network interfaces, you can configure a switch or router to forward data from one or more ports (VLANs) to a certain port, known as port mirroring, to monitor the network.

[0131] 2. Traffic metadata: Extract metadata and fingerprints from pcap packets or real-time network traffic, including but not limited to five-tuples: source IP, destination IP, source port, destination port, source MAC, destination MAC, etc.

[0132] 3. Operator model: In the actual operating environment, network security is closely related to actual business. The system's built-in algorithm rules do not meet the specific business scenarios. The role of the operator model is to customize the algorithm and execute a rule design, including built-in operators and custom operators.

[0133] 4. AI model: To meet the specific needs of network security monitoring, machine learning algorithm engineers are allowed to quickly verify the algorithm model by configuring rules without writing any program code.

[0134] 5. Rete Algorithm: This is an efficient pattern matching algorithm used to implement production rule systems. The Rete algorithm generates a network based on rule conditions, with each rule condition being a node in the network. Rete can be divided into two parts: rule compilation and runtime execution. Rule compilation is the process of generating an inference network based on a set of rules, while runtime execution is the process of feeding data into the inference network for screening.

[0135] 6. Data sink: responsible for outputting the data processed by the engine to the external system.

[0136] 7. WME: Facts or business data stored in Working Memory Element memory.

[0137] The solution of the present invention has the following advantages and characteristics:

[0138] 1) The method proposed in the present invention can be migrated and deployed in different environments and has strong operability.

[0139] 2) The experimental results of this case show that this engine has high real-time performance and strong stability.

[0140] 3) Visual rule definition is implemented, combining built-in operators with custom operators to meet the rule definition requirements of various business scenarios.

[0141] 4) Achieved seamless integration of AI computing models.

[0142] 5) This engine is completely based on network traffic metadata analysis, including information networks and industrial networks.

[0143] 6) This engine has certain requirements for computing performance and is best run in a cluster environment.

[0144] Figure 10 The framework diagram of a complex event processing system based on network traffic metadata according to an embodiment of the present invention is shown. The system includes a network metadata parsing module 1001, a rule design module 1002, a rule tree construction module 1003 and a metadata rule matching module 1004.

[0145] In a specific embodiment, the network metadata parsing module 1001 is configured to collect network traffic metadata of the switch mirror port and save it as a pcap packet, call a parsing instruction to parse the detailed information of the pcap packet to obtain a parsing result, and then save the parsing result to a Kafka message queue as a data source;

[0146] The rule design module 1002 is configured to pre-design complex event processing rules based on an operator model using a rule designer with a visual / graphical interface, wherein the complex event processing rules and corresponding processing resources are allocated according to permissions, and the allocation results are tested using an AI model, and the test results are saved as rule text;

[0147] The rule tree construction module 1003 is configured to decompose the processing rule of the complex event in the rule text into a plurality of minimum rule matching items, take each minimum rule matching item as a pattern, and then take each pattern as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree, and then saving the rete rule tree into the rule database according to the authority;

[0148] The metadata rule matching module 1004 is configured to obtain the data source from the Kafka message queue and store the data source as fact / business data in the memory, starting from the root node of the rete rule tree, matching the pattern corresponding to each type node with the fact / business data, and if the matching results are the same, outputting the results that meet the pattern corresponding to the type node to the external storage.

[0149] This system collects network traffic metadata from the switch's mirrored port and saves it as a pcap package. It then uses parsing instructions to parse the pcap package's detailed information to obtain the parsing results. Using a rule designer with a visual / graphical interface, it pre-designs complex event processing rules based on operator models, tests them using AI models, and outputs rule text. It uses a rete tree structure to store the rule text, generating a rete rule tree, which is then saved to a rule database based on permissions. Starting from the root node of the rete rule tree, it matches the pattern corresponding to each type node with the facts / business data generated by the metadata, and outputs the results that match the pattern corresponding to the type node for external storage. This system implements manually predefined business rules and integrates machine learning and artificial intelligence models, enabling multi-faceted detection of known security threats and awareness of unknown network threats.

[0150] Reference below Figure 11 , which shows a structural diagram of a computer system 1100 suitable for implementing an electronic device of an embodiment of the present application. Figure 11 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0151] like Figure 11As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the system 1100 are also stored in the RAM 1103. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0152] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, and the like; an output section 1107 including a liquid crystal display (LCD) and speakers; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or a modem. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1110 as needed, so that computer programs read therefrom can be installed in the storage section 1108 as needed.

[0153] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0154] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0155] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] The modules involved in the embodiments described in this application may be implemented in software or hardware. The units described may also be provided in a processor, and the names of these units do not, in certain circumstances, constitute limitations on the units themselves.

[0157] Embodiments of the present invention also relate to a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processor, implements the method described above. The computer program includes program code for executing the method shown in the flowchart. It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium, or any combination thereof.

[0158] The present invention collects network traffic metadata from the switch's mirrored port and saves it as a pcap package. A parsing instruction is then called to parse the detailed information of the pcap package to obtain a parsing result. A rule designer with a visual / graphical interface pre-designs complex event processing rules based on an operator model, and an AI model is used for testing and outputting rule text. A rete tree structure is used to store the rule text to obtain a rete rule tree, which is then saved to a rule database based on permissions. Starting from the root node of the rete rule tree, the pattern corresponding to each type node is matched with the facts / business data generated by the metadata, and the results that match the pattern corresponding to the type node are output to an external storage device. This method implements both manually predefined business rules and the integration of machine learning and artificial intelligence models, enabling the detection of known security threats and the perception of unknown network threats from multiple perspectives.

[0159] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A complex event processing method based on network traffic metadata, characterized in that: The following steps are involved: S1: Collecting network traffic metadata of the switch mirror port and saving it as a pcap package, calling a parsing instruction to parse detailed information of the pcap package to obtain a parsing result, and then saving the parsing result to a Kafka message queue as a data source; wherein, collecting network traffic metadata of the switch mirror port includes forwarding data of at least one port to a certain port by configuring the switch or router; the traffic metadata includes source IP, destination IP, source port, destination port, source MAC, and destination MAC; S2: Pre-design complex event processing rules based on operator models using a rule designer with a visual / graphical interface. The complex event processing rules and corresponding processing resources are allocated according to permissions. The allocation results are tested using an AI model, and the test results are saved as rule text. S3: Decomposing the processing rules of the complex events in the rule text into a number of minimum rule matching items, taking a minimum rule matching item as a pattern, and then taking a pattern as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree, and then saving the rete rule tree into the rule database according to the authority; S4: Obtain the data source from the Kafka message queue and store the data source in memory as facts / business data. Starting from the root node of the rete rule tree, match the pattern corresponding to each type node with the facts / business data. If the matching results are the same, output the result that matches the pattern corresponding to the type node to external storage; if the matching results are different, execute the following steps: Step a: passing the fact / business data to the Alpha node, detecting whether the fact / business data matches the pattern corresponding to the current Alpha node, and if so, outputting the result that matches the pattern corresponding to the current Alpha node to external storage; If not, pass the fact / business data to the next Alpha node and repeat this step. If there is no match, execute steps b and c. Step b: Pass the fact / business data to the left end of the Beta node, then encapsulate the fact / business data into a list containing only one fact / business data, and use the list as a token; Step c: Pass the fact / business data to the right end of the Beta node and the Token in the left storage area for matching; If the match is successful, the result that matches the pattern corresponding to the current Beta node is output to the external storage; If the match is unsuccessful, the fact / business data is added to the token, and the token is passed to the next node and this step is repeated; Step d: When the Token is passed to the final node, the result that meets the pattern corresponding to the final node is output to the external storage.

2. The method according to claim 1, characterized in that The step of collecting the data of the switch mirror port and saving it as a pcap package includes: Start the main service listening port configuration, then query the mirror port configuration from the database, configure the packet capture program parameters on each mirror port and collect data.

3. The method according to claim 1, characterized in that The calling parsing instruction parses the detailed information of the pcap package to obtain a parsing result, and then saves the parsing result to the Kafka message queue as a data source. The specific steps include: Monitor the Kafka message queue to determine whether the pcap package is generated; If yes, obtain the file name of the pcap package, call the parsing instruction to parse the network traffic metadata in the pcap package and write the parsing result to the Kafka metadata Topic; If not, continue to monitor the Kafka message queue.

4. The method according to claim 1, wherein The operator model specifically includes: built-in operators and custom operators.

5. The method according to claim 1, wherein The rule designer with a visual / graphical interface includes: A visual / graphical interface for defining the processing rules by mouse clicks; A visualization / graphical interface for combining multiple conditions of the processing rules and displaying them in a graphical manner.

6. The method according to claim 1, characterized in that The tests include: quick test, simulation test and Rest service test.

7. The method according to claim 1, characterized in that The processing rules include wizard rules and script rules.

8. The method according to claim 1, characterized in that The method of constructing a rete tree structure by using the pattern as a type node, thereby storing the rule text in the rete tree structure to obtain a rete rule tree, specifically comprises the following steps: Step 1: Create a root node of a rete tree structure; Step 2: Take out a pattern in the processing rule and record it as pattern i, where i is the sequence number of the pattern and i= {1,2,3…}, check the parameter type in the pattern i. If the parameter type is a new fact type, add a type node to the rete tree structure; Step 3: record the Alpha node corresponding to the pattern i as node Alpha(i), check whether the node Alpha(i) already exists, if so, record the position of the node Alpha(i), if not, add the pattern i as a new Alpha node to the rete tree structure, and then establish the Alpha memory table of the corresponding node Alpha(i) according to the pattern i; Step 4: Repeat steps 2 to 3 and increment i by 1 after each repetition until all patterns are processed; Step 5: Assemble the Beta nodes of the rete tree structure, specifically including the following steps: According to the mode i, the Beta node of the mode i is recorded as node Beta(i); When i=2, let the left input node of node Beta(2) be node Alpha(1) and the right input node be node Alpha(2); When i>2, the left input node of node Beta(i) is made into node Beta(i-1) and the right input node is made into node Alpha(i), and the memory tables of the two parent nodes of the node Beta(i) are inlined into the memory table of the node Beta(i); Step 6: Repeat step 5 until all Beta nodes are processed; Step 7: Encapsulate the action into a leaf node as the output node of the node Beta(i); Step 8: Use the final obtained rete tree structure as the rete rule tree.

9. The method according to claim 1, characterized in that The corresponding processing resources include all requested URLs.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer processor, the method according to any one of claims 1 to 9 is implemented.

11. A complex event processing system based on network traffic metadata, characterized in that: include: The network metadata parsing module is configured to collect network traffic metadata from the switch mirror port and save it as a pcap package. The parsing instruction is called to parse the detailed information of the pcap package to obtain the parsing result, and the parsing result is then saved to the Kafka message queue as a data source. The collection of network traffic metadata from the switch mirror port includes forwarding data from at least one port to a certain port by configuring the switch or router. The traffic metadata includes source IP, destination IP, source port, destination port, source MAC, and destination MAC. Rule design module: configured to pre-design complex event processing rules based on operator models using a rule designer with a visual / graphical interface, wherein the complex event processing rules and corresponding processing resources are allocated according to permissions, and the allocation results are tested using an AI model, and the test results are saved as rule text; A rule tree construction module is configured to decompose the processing rules of complex events in the rule text into a number of minimum rule matching items, take a minimum rule matching item as a pattern, and then take a pattern as a type node to construct a rete tree structure, thereby storing the rule text in the rete tree structure to obtain a rete rule tree, and then saving the rete rule tree to the rule database according to the authority; Metadata rule matching module: configured to obtain the data source from the Kafka message queue and store the data source as fact / business data in the memory, starting from the root node of the rete rule tree, matching the pattern corresponding to each type node with the fact / business data, if the matching results are the same, then output the result that matches the pattern corresponding to the type node to the external storage; if the matching results are not the same, perform the following steps: Step a: pass the fact / business data to the Alpha node, detect whether the fact / business data matches the pattern corresponding to the current Alpha node, if so, output the result that matches the pattern corresponding to the current Alpha node to the external storage; if not, pass the fact / business data to the next Alpha node and Repeat this step. If there is no match, execute step b and step c. Step b: pass the fact / business data to the left end of the Beta node, and then encapsulate the fact / business data into a list containing only one fact / business data, and use the list as a Token. Step c: pass the fact / business data to the right end of the Beta node and the Token in the left storage area for matching. If the match is successful, the result that meets the pattern corresponding to the current Beta node is output to the external storage. If the match is unsuccessful, add the fact / business data to the Token, and then pass the Token to the next node and repeat this step. Step d: When the Token is passed to the final node, the result that meets the pattern corresponding to the final node is output to the external storage.

Images