A message processing method, device, electronic device and storage medium

Abstract

The present application relates to the field of network security technology, and in particular to a message processing method, device, electronic device and storage medium, which are used to achieve multi-production and multi-consumption and improve the performance of message processing. The method includes: using a first thread group to parse NetFlow messages, obtain data flow information, and encapsulate the data flow information into a target structure, and when it is determined that there is a target PO corresponding to the target structure in the protection object PO group, the pointer set of the target structure and the target PO is added to the message slot as a write message; using a second thread group to poll the message slot, obtain the pending message, and based on the pending message, update the double linked list and the max-heap, and obtain the latest access PO and hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest access PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a message processing method, device, electronic device and storage medium. Background Art

[0002] NetFlow is a network traffic statistics protocol used to measure and count IP traffic flowing through network devices, and has become the most important standard for IP traffic analysis and statistics in the network field. The router re-encapsulates the key information of the data packet, such as source / destination address, source / destination port, protocol type, etc., into the NetFlow protocol format (currently the mainstream adopts NetFlow V9 protocol) and transmits it to the NetFlow collector on the background server. The NetFlow collector is responsible for parsing and restoring the key information. Network security personnel configure policies on the NetFlow server to detect and analyze abnormal traffic data packets, such as denial of service attack (DoS), distributed denial of service attack (DDoS), network worm virus traffic, and a large number of connection requests generated by network scanning tools. By analyzing the flow direction of the data packet, network security personnel adopt different protection and processing strategies according to different flow directions.

[0003] In actual application, network security personnel need to configure multiple protection objects (PO) on the NetFlow collector. The IP subnet and IP host protected by each PO are different. Therefore, based on the information of each configured PO, the NetFlow collector can detect and analyze whether the traffic data packets currently accessing the PO are abnormal.

[0004] In the prior art, the results of NetFlow message parsing are placed in a large table or message queue, where locking the large table will affect multi-threaded data reading and writing. At the same time, the message queue supports single producer multiconsumer (SPMC), which does not meet the actual needs of massive NetFlow message processing. In addition, querying hot spot POs and the latest access POs is slow, and frequent queries on hot spot POs and the latest access POs will also affect the efficiency of NetFlow message processing. Summary of the invention

[0005] The embodiments of the present application provide a message processing method, device, electronic device and storage medium, which realize multiple production and multiple consumption, can process a large number of NetFlow messages and quickly query the latest access PO and hot spot PO, thereby improving the performance of message processing.

[0006] The specific technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, a message processing method is provided, the method comprising:

[0008] The first thread group is used to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure. When it is determined that a target PO corresponding to the target structure exists in the protection object PO group, the pointer set of the target structure and the target PO is added to the message slot as a write message, wherein the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO;

[0009] The second thread group is used to poll the message slot to obtain the pending messages, and based on the pending messages, the double linked list and the max-heap are updated to obtain the latest accessed PO and the hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value.

[0010] In a possible embodiment, a pointer set of a target structure and a target PO is added to a message slot as a write message, including:

[0011] Calculate the hash value of the five-tuple in the target structure, and determine the slot number corresponding to the target structure based on the hash value and the total number of slots in the message slot, where the five-tuple includes: source IP, destination IP, source port, destination port, and protocol number;

[0012] When it is determined that the capacity of the target slot unit corresponding to the slot number is not full, the pointer set of the target structure and the target PO is used as a write message and added to the target slot unit using the linked list head insertion method.

[0013] Through the above method, the parsed data flow information is added to each slot unit of the message slot in a single linked list. Different from the traditional overall locking of the hash bucket, a fine-grained lock is added to each slot unit of the message slot. When a thread reads and writes the message slot, only one slot unit of the message slot is locked, and other slot units of the message slot that are read and written by other threads are not affected, thereby achieving high performance of multiple production and multiple consumption, and greatly avoiding the occurrence of multi-thread synchronization lock conflicts.

[0014] In a possible embodiment, the method further includes:

[0015] When it is determined that the capacity of the target slot unit is full, starting from the slot unit next to the target slot unit, traverse the slot units of the message slot;

[0016] If a slot unit with insufficient capacity is found during the traversal process, the message will be written and added to the slot unit with insufficient capacity using the linked list head insertion method;

[0017] If the traversal is completed and no slot unit with insufficient capacity is found, the capacity of the target slot unit will be expanded, and the written message will be added to the expanded target slot unit using the linked list head insertion method.

[0018] Through the above method, unlike the fixed length of the message queue, the depth of each slot unit of the message slot supports expansion and contraction. In extreme cases, the capacity of the target slot unit is full and no other slot unit with remaining capacity can be found. The capacity of the target slot unit can be expanded to ensure the storage of data flow information.

[0019] In a possible embodiment, based on the message to be processed, updating the double linked list and the max-heap includes:

[0020] Based on the doubly linked list position pointer of the to-be-processed PO contained in the pointer set in the to-be-processed message, move the node of the to-be-processed PO in the doubly linked list to the head node of the doubly linked list;

[0021] The access heat value of the PO to be processed is updated, and based on the access heat value and the max-heap position pointer of the PO to be processed contained in the pointer set in the message to be processed, the node position of the PO to be processed in the max-heap is updated.

[0022] Through the above method, the query of the latest accessed PO is realized. The latest accessed PO will be placed at the head of the chain, making the query of the latest accessed PO real-time. The real-time query of the heat of PO is realized, and the hot PO with the largest access heat value is pushed to the top of the stack, making the query of the hot PO real-time, with less impact on the performance of other business of the system, providing users with accurate information, and facilitating security personnel to view hot PO and the latest accessed PO and adjust security policies.

[0023] In a possible embodiment, the method further includes:

[0024] Based on the information position pointer of the PO to be processed contained in the pointer set in the message to be processed, the number of bytes and data packets contained in the structure to be processed in the message to be processed are added to the subnet counter and IP host counter corresponding to the PO to be processed.

[0025] Through the above method, it is possible to ensure that the subnet counter and host counter in each PO are accurate in counting.

[0026] In a possible embodiment, the method further includes:

[0027] The third thread group is used to perform the following operations on each PO included in the PO group:

[0028] Calculate the target values ​​of monitoring indicators of the IP subnet and IP host protected by a PO in the current time period, where the monitoring indicators include: bits per second bps and / or packets per second pps;

[0029] If the target value is greater than the corresponding monitoring indicator threshold, an alarm message is sent to the target object.

[0030] Through the above method, accurate alarm can be issued.

[0031] In a possible embodiment, calculating the target values ​​of monitoring indicators of the subnet and IP host protected by a PO in the current time period includes:

[0032] Get the initial values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the current time period;

[0033] The target value is determined based on the historical values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the previous time period, as well as the initial value and influencing factor.

[0034] Through the above method, the glitches caused by instantaneous data are suppressed, the occurrence of false alarms is greatly reduced, and the changes in traffic trends are analyzed, which makes it easier for security personnel to adjust security strategies in a timely manner according to changes in traffic trends.

[0035] In a second aspect, a message processing device is provided, the device comprising:

[0036] A parsing module, used to parse the NetFlow message by using the first thread group, obtain data flow information, and encapsulate the data flow information into a target structure, and when it is determined that there is a target PO corresponding to the target structure in the protection object PO group, add the target structure and the pointer set of the target PO as a write message to the message slot, wherein the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO;

[0037] The processing module is used to adopt the second thread group to poll the message slot, obtain the pending messages, and based on the pending messages, update the double linked list and the max-heap to obtain the latest accessed PO and the hot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot PO with the largest access heat value.

[0038] In a possible embodiment, when a pointer set of a target structure and a target PO is added to a message slot as a write message, the parsing module is further used to:

[0039] Calculate the hash value of the five-tuple in the target structure, and determine the slot number corresponding to the target structure based on the hash value and the total number of slots in the message slot, where the five-tuple includes: source IP, destination IP, source port, destination port, and protocol number;

[0040] When it is determined that the capacity of the target slot unit corresponding to the slot number is not full, the pointer set of the target structure and the target PO is used as a write message and added to the target slot unit using the linked list head insertion method.

[0041] In a possible embodiment, the parsing module is further used for:

[0042] When it is determined that the capacity of the target slot unit is full, starting from the slot unit next to the target slot unit, traverse the slot units of the message slot;

[0043] If a slot unit with insufficient capacity is found during the traversal process, the message will be written and added to the slot unit with insufficient capacity using the linked list head insertion method;

[0044] If the traversal is completed and no slot unit with insufficient capacity is found, the capacity of the target slot unit will be expanded, and the written message will be added to the expanded target slot unit using the linked list head insertion method.

[0045] In a possible embodiment, when updating the double linked list and the max-heap based on the message to be processed, the processing module is further used to:

[0046] Based on the doubly linked list position pointer of the to-be-processed PO contained in the pointer set in the to-be-processed message, move the node of the to-be-processed PO in the doubly linked list to the head node of the doubly linked list;

[0047] The access heat value of the PO to be processed is updated, and based on the access heat value and the max-heap position pointer of the PO to be processed contained in the pointer set in the message to be processed, the node position of the PO to be processed in the max-heap is updated.

[0048] In a possible embodiment, the processing module is further configured to:

[0049] Based on the information position pointer of the PO to be processed contained in the pointer set in the message to be processed, the number of bytes and data packets contained in the structure to be processed in the message to be processed are added to the subnet counter and IP host counter corresponding to the PO to be processed.

[0050] In a possible embodiment, the device further includes an alarm module, and the alarm module is used to:

[0051] The third thread group is used to perform the following operations on each PO included in the PO group:

[0052] Calculate the target values ​​of monitoring indicators of the IP subnet and IP host protected by a PO in the current time period, where the monitoring indicators include: bits per second bps and / or packets per second pps;

[0053] If the target value is greater than the corresponding monitoring indicator threshold, an alarm message is sent to the target object.

[0054] In a possible embodiment, when calculating the target values ​​of the monitoring indicators of the subnet and IP host protected by a PO in the current time period, the alarm module is further used to:

[0055] Get the initial values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the current time period;

[0056] The target value is determined based on the historical values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the previous time period, as well as the initial value and influencing factor.

[0057] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0058] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0059] In an embodiment of the present application, a first thread group is used to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure. When it is determined that a target PO corresponding to the target structure exists in the protected object PO group, a pointer set of the target structure and the target PO is added to the message slot as a write message, wherein the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO. A second thread group is used to poll the message slot to obtain pending messages, and based on the pending messages, the double linked list and the max-heap are updated to obtain the latest accessed PO and the hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value. In this way, the parsed data flow information is stored in the message slot and combined with the first thread group and the second thread group to achieve multiple production and multiple consumption, fast parsing and fast calculation, and can process a large number of NetFlow messages. At the same time, the hot spot PO is pushed to the top of the heap through the max-heap, and the hot spot PO is displayed in real time and accurately, avoiding the traversal query operation of all POs, and has little impact on the performance of other services in the system. The latest accessed PO is placed at the head of the chain through a double linked list, and the recently accessed PO is displayed in real time and accurately, avoiding the traversal query operation of all POs, and has little impact on the performance of other services in the system, providing users with accurate information, thereby improving the performance of message processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A diagram of possible application scenarios in the embodiments of the present application;

[0061] Figure 2 A flowchart of a message processing method provided in an embodiment of the present application;

[0062] Figure 3 This is a schematic diagram of a message slot in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of obtaining the latest visited PO and hot spot PO in an embodiment of the present application;

[0064] Figure 5 Another flowchart of the message processing method in the embodiment of the present application is shown;

[0065] Figure 6 A schematic diagram of the structure of a message processing device provided in an embodiment of the present application;

[0066] Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0068] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0069] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0070] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0071] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0072] NetFlow: A protocol proposed by Cisco to describe connection information passing through routers. Netflow has multiple versions, the most commonly used ones are NetFlow v5, NetFlow v9, and NetfFlow v10. By parsing these protocols, the five-tuple (source IP, destination IP, source port, destination port, protocol number), number of packets, number of bytes, and other information of the current data flow can be obtained.

[0073] The least recently used (LRU) algorithm is a commonly used cache eviction strategy, mainly used to manage caches in computer systems. Its core idea is to prioritize the least recently used data to ensure that the cache always retains the data that is most likely to be accessed in the near future.

[0074] Least frequently used (LFU) algorithm: is a common cache eviction strategy, which is mainly used to manage data in the cache and ensure that the least frequently used data is eliminated first.

[0075] Exponential moving average (EMA): Similar to weighted moving average, except that the weights of the values ​​decrease exponentially rather than linearly.

[0076] Protection object (PO): Each PO protects different IP subnets, IP hosts, and monitoring indicator thresholds. A PO contains: id, name, counter counter of ip4 subnet, bits per second bps and packets per second pps, counter, bps and pps of ip4 host, counter, bps and pps of ip6 subnet, counter, bps and pps of ip6 host, and monitoring indicator threshold. The counter records the total number of bytes and packets in a time period, and is cleared after a time period is calculated.

[0077] The following is a brief introduction to the design concept of the embodiment of the present application:

[0078] NetFlow is a network traffic statistics protocol used to measure and count IP traffic flowing through network devices, and has become the most important standard for IP traffic analysis and statistics in the network field. The router re-encapsulates the key information of the data packet, such as source / destination address, source / destination port, protocol type, etc., into the NetFlow protocol format (currently the mainstream adopts NetFlow V9 protocol) and transmits it to the NetFlow collector on the background server. The NetFlow collector is responsible for parsing and restoring the key information. Network security personnel configure policies on the NetFlow server to detect and analyze abnormal traffic data packets, such as denial of service attack (DoS), distributed denial of service attack (DDoS), network worm virus traffic, and a large number of connection requests generated by network scanning tools. By analyzing the flow direction of the data packet, network security personnel adopt different protection and processing strategies according to different flow directions.

[0079] In actual application, network security personnel need to configure multiple POs on the NetFlow collector. The IP subnet and IP host protected by each PO are different. Therefore, based on the information of each configured PO, the NetFlow collector can detect and analyze whether the traffic data packets currently accessing the PO are abnormal.

[0080] In the prior art, the results of NetFlow message parsing are placed in a large table or message queue, where locking the large table will affect multi-threaded data reading and writing. At the same time, the message queue supports single producer multiconsumer (SPMC), and the message queue has a fixed queue, which does not meet the actual needs of massive NetFlow message processing. At the same time, the protected IP subnet information is statically loaded to achieve protected IP subnet monitoring, detection and alarm, but does not support dynamic protection configuration, resulting in inflexible use, and the inability to reflect the accessed hotspot PO and the latest accessed PO in real time, resulting in poor user experience. Frequent queries on the hotspot PO and the latest accessed PO will also affect the NetFlow message processing efficiency.

[0081] For example, the open source software fastnetmon must be statically configured with monitored IP subnet information and thresholds before startup. It does not support loading protection information when the software is running. In addition, the software uses the user datagram protocol (UDP) server of the traditional protocol stack, and the processing of NetFlow parsing and statistics is completed by a single core and single thread. This architecture design cannot meet the processing requirements of large-volume NetFlow messages, resulting in untimely monitoring and network attacks on protected services.

[0082] In addition, when judging whether a PO is under attack, the values ​​of monitoring indicators (bits per second bps and / or packets per second pps) calculated using traditional calculation methods are prone to showing a large number of glitches on the traffic trend graph, which can lead to a large number of false alarms.

[0083] In view of this, in an embodiment of the present application, a message processing method, device, electronic device and storage medium are provided, and a first thread group is used to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure, and when it is determined that a target PO corresponding to the target structure exists in the protection object PO group, a pointer set of the target structure and the target PO is added to the message slot as a write message, wherein the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO; at the same time, a second thread group is used to poll the message slot to obtain a pending message, and based on the pending message, the double linked list and the max-heap are updated to obtain the latest accessed PO and the hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value. In this way, by storing the parsed data flow information through the message slot, combined with the first thread group and the second thread group, multiple production and multiple consumption are realized, fast parsing and fast calculation are realized, and a large number of NetFlow messages can be processed. At the same time, the hot spot PO is pushed to the top of the heap through the big top heap, and the hot spot PO is displayed in real time and accurately, avoiding the traversal query operation of all POs, which has little impact on the performance of other services of the system. The latest access PO is placed at the head of the chain through the double linked list, and the latest access PO is displayed in real time and accurately, avoiding the traversal query operation of all POs, which has little impact on the performance of other services of the system, providing users with accurate information, thereby improving the performance of message processing. In addition, it supports dynamic protection configuration, and can add POs to the PO group when the program is running, which improves the flexibility of configuration; the EMA method is used to calculate the target value of the monitoring indicator, so that the flow change tends to be flat, and prevents false alarms caused by normal instantaneous large flow access.

[0084] See also Figure 1As shown, a possible application scenario diagram in an embodiment of the present application includes a first thread group 110 , a second thread group 120 , a third thread group 130 , a hash table 140 , a message slot 150 , and a PO group 160 .

[0085] The first thread group 110 is a message parsing thread group, which is used to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure. Based on the target IP contained in the five-tuple in the target structure, the hash table 140 is queried to determine whether there is a target PO corresponding to the five-tuple in the PO group. When the target PO exists, the pointer set of the target structure and the target PO is added to the message slot as a write message.

[0086] The second thread group 120 is a polling receiving thread group, which is used to poll the message slot, obtain the pending messages, and update the double linked list and the big top heap based on the pending messages, that is, update the latest access PO and hot spot PO, obtain the latest access PO and hot spot PO in the PO group, and update the subnet counter and host counter of the pending PO corresponding to the pending message based on the pending message. In addition, it is worth noting that in the embodiment of the present application, the ratio of the number of threads of the first thread group and the number of threads of the second thread group can be adjusted to optimize the message parsing and message processing performance.

[0087] The third thread group 130 is an indicator value calculation thread group, which is used to perform the following operations on each PO included in the PO group: calculate the target value of the monitoring indicator of the IP subnet and IP host protected by a PO in the current time period, where the monitoring indicators include: bits per second bps and / or packets per second pps; if the target value is greater than the corresponding monitoring indicator threshold, send an alarm message to the target object.

[0088] The hash table 140 is used to store the IP subnet and pointer set of each PO contained in the PO group. The pointer set includes: a double-linked list position pointer, a max-heap position pointer and an information position pointer of the corresponding PO.

[0089] The message slot 150 is used to store each message, wherein each message includes: a structure corresponding to the corresponding data flow information and a pointer set of POs corresponding to the data flow information, and the structure records information such as a five-tuple (source IP, destination IP, source port, destination port, protocol number), number of packets, and number of bytes. In the embodiment of the present application, the total number of slots of the message slot 150 and the depth of each slot can be pre-allocated by pre-evaluating the size of the message flow.

[0090] PO group 160 includes various POs, wherein each PO includes: id, name, counter, bps and pps of ip4 subnet, counter, bps and pps of ip4 host, counter, bps and pps of ip6 subnet, counter, bps and pps of ip6 host, and monitoring index threshold. In the embodiment of the present application, POs in the PO group can be dynamically added during the program running process, and there is no limit on the number of POs in the PO group.

[0091] See also Figure 2 As shown, it is an implementation flow chart of a message processing method provided in an embodiment of the present application. The specific implementation process of the method is as follows

[0092] Step 20: Use the first thread group to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure. When it is determined that there is a target PO corresponding to the target structure in the protection object PO group, the pointer set of the target structure and the target PO is added to the message slot as a write message.

[0093] Among them, the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO, and the pointer set of the target PO also includes: an information position pointer of the target PO, and the information position pointer is used to indicate the position of the target PO in the PO group.

[0094] In an embodiment of the present application, a first thread group is used to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure. At the same time, the five-tuple in the target structure is taken out to determine whether there is a target PO corresponding to the target structure in the protection object PO group. If it is determined that there is a target PO corresponding to the target structure in the protection object PO group, the pointer set of the target structure and the target PO is used as a write message and added to the message slot. If it is determined that there is no target PO corresponding to the target structure in the protection object PO group, the target structure is discarded.

[0095] The five-tuple includes: source IP, destination IP, source port, destination port, and protocol number.

[0096] Specifically, in an embodiment of the present application, based on the target IP in the quintuple, a hash table is queried to determine whether there is a target PO corresponding to the protected IP subnet and the target IP in the PO group. If so, it is determined that there is a target PO corresponding to the target structure in the protected object PO group. If not, it is determined that there is no target PO corresponding to the target structure in the protected object PO group.

[0097] The hash table is used to store the IP subnet and pointer set protected by each PO contained in the PO group, the key is the protected IP subnet, and the value is the pointer set.

[0098] Optionally, in an embodiment of the present application, a possible embodiment is provided for adding a pointer set of a target structure and a target PO as a write message to a message slot, and the specific steps are as follows:

[0099] Step 200: Calculate the hash value of the five-tuple in the target structure, and determine the slot number corresponding to the target structure based on the hash value and the total number of slots of the message slot.

[0100] In the embodiment of the present application, a hash function is used to calculate the hash value of the five-tuple in the target structure, and the hash value is used to take the remainder of the total number of slots of the message slot to obtain the slot number corresponding to the target structure.

[0101] The slot number corresponding to the target structure can be expressed as: nMessageSlot=hash mod N, hash is the hash value of the five-tuple in the target structure, and N is the total number of message slots.

[0102] Step 201: When it is determined that the capacity of the target slot unit corresponding to the slot number is not full, the pointer set of the target structure and the target PO is used as a write message and added to the target slot unit by using the linked list head insertion method.

[0103] In an embodiment of the present application, when it is determined that the capacity of the target slot unit corresponding to the slot number is not full, the hash table is queried to obtain a pointer set of the target PO, and the target structure and the pointer set of the target PO are used as write messages and added to the target slot unit using the linked list head insertion method.

[0104] Among them, each slot unit in each message slot has a slot_size variable and a slot_capacity variable. The slot_size variable records the capacity used by the slot unit, and the slot_capacity variable records the depth of the slot unit. These two variables are atomic operations.

[0105] Specifically, in the embodiment of the present application, it is determined whether the capacity of the target slot unit is not full based on the used capacity of the target slot unit and the depth of the target slot unit.

[0106] In this way, the parsed data flow information is added to each slot unit of the message slot through a single linked list. Different from the traditional overall locking of the hash bucket, a fine-grained lock is added to each slot unit of the message slot. When a thread reads and writes the message slot, only one slot unit of the message slot is locked, and other slot units of the message slot that are read and written by other threads are not affected, thus achieving high performance of multi-producer multi-consumer (MPMC) and greatly avoiding the occurrence of multi-thread synchronization lock conflicts.

[0107] Step 202: When it is determined that the capacity of the target slot unit is full, traverse the slot units of the message slot starting from the slot unit next to the target slot unit.

[0108] In the embodiment of the present application, when it is determined that the capacity of the target slot unit is full, nMessageSlot is incremented by 1, and the slot units of the message slot are traversed starting from the slot unit next to the target slot unit. The results of the traversal include the following two results:

[0109] Result 1: During the traversal process, a slot unit with insufficient capacity is found.

[0110] In the embodiment of the present application, if a slot unit with insufficient capacity is found during the traversal process, the written message will be added to the slot unit with insufficient capacity by using the linked list head insertion method.

[0111] Result 2: The traversal is completed and no slot unit with insufficient capacity is found.

[0112] In the embodiment of the present application, if the traversal is completed and no slot unit with insufficient capacity is found, the capacity of the target slot unit is expanded, and the written message is added to the expanded target slot unit using the linked list header insertion method.

[0113] The capacity expansion may be doubled, which is not limited in the embodiments of the present application.

[0114] In this way, unlike the fixed length of the message queue, the depth of each slot unit of the message slot supports expansion and contraction. In extreme cases, the capacity of the target slot unit is full and no other slot units with remaining capacity can be found. The capacity of the target slot unit can be expanded to ensure the storage of data flow information.

[0115] For example, see Figure 3 As shown, it is a schematic diagram of the message slot in the embodiment of the present application. The used capacity slot_size of the slot unit with slot number 0 is 4, and the depth slot_capacity is 8. The slot unit with slot number 1 is expanded. The used capacity slot_size of the slot unit with slot number 1 is 10, and the depth slot_capacity is 16. The structure and pointer set are added to the corresponding slot unit by using the linked list head insertion method.

[0116] Step 21: Use the second thread group to poll the message slot to obtain the pending messages, and based on the pending messages, update the double linked list and the max-heap to obtain the latest accessed PO and hot spot PO in the PO group.

[0117] The pending message includes: a pending structure and a corresponding pointer set, and the corresponding pointer set includes: a double-linked list position pointer, a max-heap position pointer, and an information position pointer of the pending PO. The information position pointer of the pending PO is used to indicate the position of the pending PO in the PO group. The PO corresponding to the head node in the double-linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hotspot PO with the largest access heat value. The order of nodes in the double-linked list is sorted by access time, and the order of nodes in the max-heap is sorted by access heat value.

[0118] Optionally, in an embodiment of the present application, a possible embodiment is provided to update a double linked list and a max-heap based on the pending message to obtain the latest accessed PO and hot spot PO in the PO group, and the specific steps are as follows:

[0119] Step 210: Based on the doubly linked list position pointer of the PO to be processed contained in the pointer set in the message to be processed, move the node of the PO to be processed in the doubly linked list to the head node of the doubly linked list.

[0120] In the embodiment of the present application, the LRU algorithm is adopted to determine the node position of the pending PO in the double linked list based on the double linked list position pointer of the pending PO contained in the pointer set of the pending message, and move the node of the pending PO in the double linked list to the head node of the double linked list. At this time, the head node in the double linked list can be queried to obtain the latest access PO.

[0121] In this way, the query of the most recently visited PO is realized. Using the hash table and the double linked list, the most recently visited PO will be placed at the head of the chain, making the query of the most recently visited PO real-time, with less impact on the performance of other business of the system, and providing users with accurate information.

[0122] Step 211: Update the access heat value of the PO to be processed, and update the node position of the PO to be processed in the max-heap based on the access heat value and the max-heap position pointer of the PO to be processed contained in the pointer set in the message to be processed.

[0123] In an embodiment of the present application, the LFU algorithm is used to update the access heat value of the PO to be processed, and the node position of the PO to be processed in the big heap is determined based on the big heap position pointer of the PO to be processed contained in the pointer set in the message to be processed. The node position of the PO to be processed in the big heap is updated based on the access heat value of the PO to be processed. If the access heat value of the PO to be processed is the largest, the node position of the PO to be processed will be updated to the vertex.

[0124] In this way, real-time query of PO popularity is realized. By using hash tables and max-heaps, POs with the largest access popularity values ​​are pushed to the top of the heap, making queries of hot POs real-time with less impact on other business performance of the system. This provides users with accurate information and facilitates security personnel to view hot POs and adjust security policies.

[0125] Exemplary, Participatory Figure 4 As shown, it is a schematic diagram of obtaining the latest access PO and hot spot PO in an embodiment of the present application. The head node in the double linked list stores the latest access PO, and the vertex in the max-heap stores the hot spot PO with the largest access heat value. At this time, the latest access PO is PO[0], and the hot spot access PO is PO[1].

[0126] Optionally, in an embodiment of the present application, after obtaining the message to be processed, based on the information location pointer of the PO to be processed contained in the pointer set in the message to be processed, the number of bytes and the number of data packets contained in the structure to be processed in the message to be processed are added to the subnet counter and host counter corresponding to the PO to be processed.

[0127] Among them, the subnet counter corresponding to the PO to be processed is the counter of the IP subnet corresponding to the target IP in the quintuple contained in the structure to be processed, and the host counter corresponding to the PO to be processed is the counter of the IP host corresponding to the target IP in the quintuple contained in the structure to be processed. The counters are used to count the number of bytes and the number of data packets respectively.

[0128] In this way, the subnet counter and host counter in each PO can be guaranteed to be accurate.

[0129] Further, in the embodiment of the present application, a third thread group is used to monitor and alarm each PO included in the PO group. Specifically, the third thread group is used to perform the following operations on each PO included in the PO group:

[0130] Step A1: Calculate the target values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the current time period.

[0131] The monitoring indicators include: bits per second (bps) and / or packets per second (pps).

[0132] In the embodiment of the present application, the initial value of the monitoring index of an IP subnet protected by a PO in the current time period is obtained, and then the EMA algorithm is used to determine the target value of the monitoring index of the IP subnet in the current time period according to the historical value of the monitoring index of the IP subnet in the previous time period, as well as the initial value and the influencing factor. The initial value of the monitoring index of an IP host protected by a PO in the current time period is obtained, and then the target value of the monitoring index of the IP host in the current time period is determined according to the historical value of the monitoring index of the IP host in the previous time period, as well as the initial value and the influencing factor.

[0133] Among them, the calculation formula 1 of the target value of the monitoring index of the IP subnet or IP host protected by a PO is Vt=βVt-1+(1-β)Xt, t≥1. Under the initial condition, when t=0, Vt=0, Vt-1 represents the calculation result of the target value of the monitoring index of the IP subnet or IP host protected by a PO in the previous time period, that is, the historical value, β represents the influence factor, which is usually between 0 and 1. The closer β is to 1, the stronger the ability to suppress instantaneous spike data and the better the smoothness. On the contrary, the closer β is to 0, the better the real-time performance. Xt represents the initial value of the monitoring index of the IP subnet or IP host protected by a PO in the current time period.

[0134] In addition, it is worth noting that in the embodiment of the present application, the total number of bytes in the subnet counter corresponding to the IP subnet in the current time period is calculated and divided by the value of the length of the time period to obtain the bps initial value of the IP subnet, the total number of data packets in the subnet counter corresponding to the IP subnet in the current time period is calculated and divided by the value of the length of the time period to obtain the pps initial value of the IP subnet, the total number of bytes in the host counter corresponding to the IP host in the current time period is calculated and divided by the value of the length of the time period to obtain the bps initial value of the IP host, the total number of data packets in the host counter corresponding to the IP host in the current time period is calculated and divided by the value of the length of the time period to obtain the pps initial value of the IP host.

[0135] In this way, the EMA algorithm is used to calculate the target value of the monitoring indicator, which suppresses the glitches caused by instantaneous data, greatly reduces the occurrence of false alarms, and analyzes changes in traffic trends, making it easier for security personnel to adjust security strategies in a timely manner based on changes in traffic trends.

[0136] In order to improve the computer computing performance, the above calculation formula 1 is transformed, the common coefficient β is merged and written as β(Vt-1-Xt), the coefficient 1 in front of Xt is extracted and written as Xt alone, and the transformed formula 2 is as follows: Vt=Xt+β(Vt-1-Xt), t>0.

[0137] In comparison, Formula 2 is simpler than Formula 1 and reduces one coefficient multiplication operation, thus improving CPU usage. The performance comparison between Formula 1 and Formula 2 is shown in Table 1.

[0138] Table 1

[0139]

[0140] As can be seen from Table 1, when executed 1 million times, the computing performance of Formula 2 is improved by 223% compared with Formula 1. Therefore, Formula 2 has more engineering application value and can reduce CPU computing time.

[0141] Step A2: If the target value is greater than the corresponding monitoring indicator threshold, an alarm message is sent to the target object.

[0142] Among them, the IP subnet and IP host protected by each PO have their own corresponding monitoring indicator thresholds, and the target object can be network security personnel, which is not limited in the embodiments of the present application.

[0143] In the embodiment of the present application, if the target value is greater than the corresponding monitoring indicator threshold, an alarm message is generated and sent to the target object, wherein the alarm message indicates that the corresponding PO has detected an abnormality.

[0144] Based on the above embodiments, see Figure 5 FIG. 1 is another flow chart of a message processing method in an embodiment of the present application, which specifically includes:

[0145] Step 50: Use the first thread group to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure.

[0146] Step 51: through the hash table and the quintuple of the target structure, determine whether there is a target PO corresponding to the target structure in the PO group. If so, execute step 52; otherwise, execute step 50.

[0147] Step 52: Add the pointer set of the target structure and the target PO as a write message to the message slot.

[0148] Step 53: Use the second thread group to determine whether there is a message in the message slot. If so, execute step 54; otherwise, execute step 53.

[0149] Step 54: Based on the obtained message, update the double linked list and the max-heap, obtain the latest access PO and hot spot PO in the PO group, and add the number of bytes and data packets contained in the structure in the waiting message to the subnet counter and IP host counter corresponding to the corresponding PO.

[0150] Step 55: Use the third thread group to calculate the target values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the current time period.

[0151] Step 56: Determine whether the target value is greater than the corresponding monitoring indicator threshold. If so, execute step 57; otherwise, execute step 55.

[0152] Step 57: Generate alarm information and send the alarm information to the target object.

[0153] Based on the same inventive concept, the present application embodiment provides a message processing device, please refer to Figure 6 , the device comprises:

[0154] The parsing module 601 is used to parse the NetFlow message by using the first thread group, obtain data flow information, and encapsulate the data flow information into a target structure, and when it is determined that there is a target PO corresponding to the target structure in the protection object PO group, add the pointer set of the target structure and the target PO as a write message to the message slot, wherein the pointer set of the target PO includes: a double-linked list position pointer and a max-heap position pointer of the target PO;

[0155] Processing module 602 is used to adopt the second thread group to poll the message slot, obtain the pending messages, and update the double linked list and the max-heap based on the pending messages to obtain the latest accessed PO and the hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value.

[0156] In a possible embodiment, when the pointer set of the target structure and the target PO is added to the message slot as a write message, the parsing module 601 is further used to:

[0157] Calculate the hash value of the five-tuple in the target structure, and determine the slot number corresponding to the target structure based on the hash value and the total number of slots in the message slot, where the five-tuple includes: source IP, destination IP, source port, destination port, and protocol number;

[0158] When it is determined that the capacity of the target slot unit corresponding to the slot number is not full, the pointer set of the target structure and the target PO is used as a write message and added to the target slot unit using the linked list head insertion method.

[0159] In a possible embodiment, the parsing module 601 is further used for:

[0160] When it is determined that the capacity of the target slot unit is full, starting from the slot unit next to the target slot unit, traverse the slot units of the message slot;

[0161] If a slot unit with insufficient capacity is found during the traversal process, the message will be written and added to the slot unit with insufficient capacity using the linked list head insertion method;

[0162] If the traversal is completed and no slot unit with insufficient capacity is found, the capacity of the target slot unit will be expanded, and the written message will be added to the expanded target slot unit using the linked list head insertion method.

[0163] In a possible embodiment, when updating the double linked list and the max-heap based on the message to be processed, the processing module 602 is further used to:

[0164] Based on the doubly linked list position pointer of the to-be-processed PO contained in the pointer set in the to-be-processed message, move the node of the to-be-processed PO in the doubly linked list to the head node of the doubly linked list;

[0165] The access heat value of the PO to be processed is updated, and based on the access heat value and the max-heap position pointer of the PO to be processed contained in the pointer set in the message to be processed, the node position of the PO to be processed in the max-heap is updated.

[0166] In a possible embodiment, the processing module 602 is further configured to:

[0167] Based on the information position pointer of the PO to be processed contained in the pointer set in the message to be processed, the number of bytes and data packets contained in the structure to be processed in the message to be processed are added to the subnet counter and IP host counter corresponding to the PO to be processed.

[0168] In a possible embodiment, the device further includes an alarm module 603, and the alarm module 603 is used to:

[0169] The third thread group is used to perform the following operations on each PO included in the PO group:

[0170] Calculate the target values ​​of monitoring indicators of the IP subnet and IP host protected by a PO in the current time period, where the monitoring indicators include: bits per second bps and / or packets per second pps;

[0171] If the target value is greater than the corresponding monitoring indicator threshold, an alarm message is sent to the target object.

[0172] In a possible embodiment, when calculating the target values ​​of the monitoring indicators of the subnet and IP host protected by a PO in the current time period, the alarm module 603 is further used to:

[0173] Get the initial values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the current time period;

[0174] The target value is determined based on the historical values ​​of the monitoring indicators of the IP subnet and IP host protected by a PO in the previous time period, as well as the initial value and influencing factor.

[0175] Based on the above embodiments, see Figure 7 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0176] An embodiment of the present application provides an electronic device, which may include a processor 710 (Center Processing Unit, CPU), a memory 720, an input device 730 and an output device 740, etc. The input device 730 may include a keyboard, a mouse, a touch screen, etc., and the output device 740 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0177] The memory 720 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 710 with program instructions and data stored in the memory 720. In the embodiment of the present application, the memory 720 may be used to store the program of any message processing method in the embodiment of the present application.

[0178] The processor 710 calls the program instructions stored in the memory 720, and the processor 710 is used to execute any message processing method in the embodiments of the present application according to the obtained program instructions.

[0179] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the message processing method in any of the above method embodiments is implemented.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0184] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A message processing method, characterized in that: include: A first thread group is used to parse the NetFlow message, obtain data flow information, and encapsulate the data flow information into a target structure, and when it is determined that a target PO corresponding to the target structure exists in the protection object PO group, the target structure and the pointer set of the target PO are added to the message slot as a write message, wherein the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO; A second thread group is used to poll the message slot to obtain pending messages, and based on the pending messages, a double linked list and a max-heap are updated to obtain the latest accessed PO and hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value.

2. The method according to claim 1, characterized in that The step of adding the target structure and the pointer set of the target PO as a write message to the message slot includes: Calculate the hash value of the five-tuple in the target structure, and determine the slot number corresponding to the target structure based on the hash value and the total number of slots of the message slot, wherein the five-tuple includes: source IP, destination IP, source port, destination port, and protocol number; When it is determined that the capacity of the target slot unit corresponding to the slot number is not full, the pointer set of the target structure and the target PO is used as a write message and added to the target slot unit by using the linked list head insertion method.

3. The method according to claim 2, characterized in that The method further comprises: When it is determined that the capacity of the target slot unit is full, starting from the slot unit next to the target slot unit, traversing each slot unit of the message slot; If a slot unit with insufficient capacity is found during the traversal process, the written message is added to the slot unit with insufficient capacity by using the linked list head insertion method; If the traversal is completed and no slot unit with insufficient capacity is found, the capacity of the target slot unit is expanded, and the write message is added to the expanded target slot unit by using the linked list header insertion method.

4. The method according to claim 1, characterized in that: The updating of the double linked list and the max-heap based on the to-be-processed message comprises: Based on the double-linked list position pointer of the to-be-processed PO contained in the pointer set in the to-be-processed message, move the node of the to-be-processed PO in the double-linked list to the head node of the double-linked list; The access heat value of the pending PO is updated, and based on the access heat value and the max-heap position pointer of the pending PO contained in the pointer set in the pending message, the node position of the pending PO in the max-heap is updated.

5. The method according to claim 1, characterized in that: The method further comprises: Based on the information position pointer of the PO to be processed contained in the pointer set in the message to be processed, the number of bytes and data packets contained in the structure to be processed in the message to be processed are added to the subnet counter and IP host counter corresponding to the PO to be processed.

6. The method according to claim 1, characterized in that The method further comprises: The third thread group is used to perform the following operations on each PO included in the PO group: Calculate the target values ​​of monitoring indicators of the IP subnet and IP host protected by a PO in the current time period, wherein the monitoring indicators include: bits per second bps and / or packets per second pps; If the target value is greater than the corresponding monitoring indicator threshold, an alarm message is sent to the target object.

7. The method according to claim 6, characterized in that The calculation of the target values ​​of the monitoring indicators of the subnet and IP host protected by a PO in the current time period includes: Obtaining the initial values ​​of the monitoring indicators of the IP subnet and IP host protected by the one PO in the current time period; The target value is determined based on the historical values ​​of the monitoring indicators of the IP subnet and IP host protected by the PO in the previous time period, as well as the initial value and the influencing factor.

8. A message processing device, characterized in that: include: A parsing module, used to parse the NetFlow message by using the first thread group, obtain data flow information, and encapsulate the data flow information into a target structure, and when it is determined that a target PO corresponding to the target structure exists in the protection object PO group, add the target structure and the pointer set of the target PO as a write message to the message slot, wherein the pointer set of the target PO includes: a double linked list position pointer and a max-heap position pointer of the target PO; A processing module is used to adopt a second thread group to poll the message slot, obtain the pending messages, and based on the pending messages, update the double linked list and the max-heap to obtain the latest accessed PO and the hot spot PO in the PO group, wherein the PO corresponding to the head node in the double linked list is the latest accessed PO, and the PO corresponding to the vertex in the max-heap is the hot spot PO with the largest access heat value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images