Processing system and processing method for multi-thread convergence time sequence data

Abstract

The invention relates to the technical field of computer security, in particular to a processing system and a processing method for multi-thread aggregation time series data, which save a large amount of disk space by not performing log storage but only performing log cache, so that the problem that data accuracy needs to be verified in a later retrieval mode is avoided, and the data accuracy is improved. The data required by the rule is only reserved within a certain time, so that the performance of later retrieval and storage is omitted; the timer can be adjusted according to actual requirements, so that the data accuracy is ensured and the alarm delay is shortened within a certain range; calculation in the method is basically based on the number of threads, the time complexity is O (1), and the performance later retrieval mode is much higher; data corresponding processing is carried out among the threads in the whole process, and the performance can be linearly improved only by increasing the number of the threads within a certain range.

Description

Technical Field

[0001] The present invention relates to the field of computer security technology, and in particular to a processing system and method for multi-threaded converged time series data. Background Art

[0002] With the continuous development of information technology, network traffic has shown explosive growth, especially driven by technologies such as 5G, cloud computing, and the Internet of Things, network traffic has become more complex and diverse. This requires the network traffic alarm system to be able to process and analyze large-scale data flows to identify and respond to potential security threats.

[0003] Network attack methods are becoming increasingly complex, including APT attacks, 0day vulnerability exploits, DDoS attacks, etc. These attacks are often difficult to detect through traditional security equipment. Therefore, the network traffic alarm system needs to have advanced analysis capabilities to detect these complex threats in a timely manner. Full-flow data collection is the basis of network traffic alarms, which requires the system to monitor all data packets in the network in real time and conduct in-depth analysis of these data. This includes technologies such as session reassembly, application layer protocol parsing, and pcap data storage to ensure the integrity and accuracy of the data. In the massive amount of network traffic data, how to quickly and accurately identify abnormal behavior and potential threats is a challenge.

[0004] In order to adapt to the huge traffic and the requirements of rapid identification, the full-traffic data collection system generally strips away other services and focuses only on protocol analysis and log output. If you want to find the alarm, you will generally not add it to the identification process.

[0005] Due to memory space limitations and timeliness, the full-flow collection system will inevitably have the problem of threshold judgment for log output. Generally, the threshold judgment for log output will be combined with time and space. For example, when two people are talking on the phone, the call may be very long and the flow may not end when doing flow restoration logs. Under the condition of space limitations and timeliness, the same flow will output the existing logs first after a certain space or a period of time, and subsequent logs will continue to be collected and output according to threshold conditions.

[0006] Since the output is generally multi-threaded, under such conditions, the output log will have several characteristics:

[0007] 1. After the system time is full, the logs before the full time may be output;

[0008] 2. There is an inconsistency in the order of multiple logs;

[0009] 3. The time when the traffic stops and the time when the last log is pushed may be quite different.

[0010] A Chinese patent with publication number CN116610252A discloses a big data acquisition system with data summarization and storage functions. When analyzing such logs, most of them are based on the stored logs, and the results are obtained by retrieving and extracting the logs and analyzing them. Although this method can ignore the problem of output log order, it is not easy to balance the accuracy and timeliness of the data. For example, when judging the traffic threshold alarm at 11 o'clock, if the data is accurate, it is necessary to ensure that the data at 11 o'clock is stored in place, and no new data at 11 o'clock will be generated after the query. It is difficult to grasp this moment. If the moment problem is ignored, the subsequent compensation mechanism will involve the need to modify the previous data, high complexity, and low performance.

[0011] Furthermore, existing technologies basically directly store unordered logs, relying on the time attributes of the data itself and the sorting capabilities of the database to retrieve and extract the logs and obtain the results after analysis. In order to ensure that the final results are accurate, there are several difficult problems to solve:

[0012] 1. How to ensure that the logs of scheduled scans are complete. For example, if the rule is to compare the traffic changes of a specified IP within two consecutive hours and the changes exceed a certain range, in this scenario, it is necessary to ensure that the corresponding logs of the two consecutive hours have been stored. The usual practice is to use transaction storage to record the time or to try to read multiple times to observe the changes in the number of logs. The use of transaction methods will greatly reduce storage performance, and the method of multiple attempts to read is not stable enough and can only ensure the integrity of the logs to a certain extent.

[0013] 2. How to access stored log data with high performance when the data volume is large. In scenarios with many rules, each rule may correspond to logs with different structures. One table can be used to store the data of all rules. However, when the data volume is large, the data required for each rule retrieval will be very performance-intensive. Even if the data is partitioned by rules and time, there will be too many partitions and performance consumption for partition retrieval. Summary of the invention

[0014] In view of the above problems, the present invention provides a processing system and method for multi-threaded converged time series data.

[0015] The technical solution adopted is a multi-threaded time series data processing system, including a rule matching component and a data cache component;

[0016] The rule matching component is used to connect to network traffic and extract protocol data in the traffic based on set rules;

[0017] The data cache component is used to cache the ordered protocol data that has been aggregated, and provide a first-in-first-out reading interface.

[0018] Optionally, the rule matching component includes a rule matching thread group and a data aggregation thread group;

[0019] The rule matching thread group includes multiple rules, filters target data from the received network traffic data based on the set rules, and pushes the target data to the data aggregation thread group;

[0020] The data aggregation thread group has the same number of threads as the rule matching thread group, and the threads in the data aggregation thread group correspond one-to-one to the threads in the rule matching thread group.

[0021] Optionally, the data in each thread in the data aggregation thread group is stored in a linked list, and the content of each data is aggregated in the linked list according to a regular time granularity.

[0022] Optionally, after the data aggregation thread group receives the target data pushed by the rule matching thread group, it will pre-insert the data in a thread, and if the data at the tail of the linked list in the thread is smaller than the pre-inserted data, new tail node data will be added. If the pre-inserted data is consistent with the data at the tail of the linked list, the corresponding data will be aggregated.

[0023] Optionally, a timer is set in the data aggregation thread group, and within each timer cycle, the data with the smallest time in the linked list header data of all data aggregation threads is taken. If there are multiple identical times, the first one is taken. While judging the minimum value, each linked list header data that is equal to the minimum value is marked. When the judgment is completed, all minimum values ​​are disconnected and pushed to the data cache component.

[0024] Optionally, each of the cache queues matches the rules in the rule matching thread group, and the same batch of rules will only be pushed to the same cache queue.

[0025] The present application also provides a method for processing multi-threaded converged time series data, which is implemented based on a multi-threaded converged time series data processing system, wherein the method is applied to a rule matching thread group end, and the method includes the following steps:

[0026] Receive network traffic data;

[0027] Filter target data from network traffic data based on set rules;

[0028] Push the target data to the data aggregation thread group.

[0029] The present application also provides a method for processing multi-threaded aggregated time series data, which is implemented based on a multi-threaded aggregated time series data processing system, wherein the method is applied to a data aggregation thread group end, and the method includes the following steps:

[0030] Create a linked list and aggregate the content of each piece of data into the linked list according to the time granularity of the set rules;

[0031] Receive the target data pushed by the rule-matching thread group and pre-insert it into the linked list;

[0032] Within a timer cycle, the data with the smallest time in the linked list header data of all data aggregation threads is disconnected and pushed to the data cache component.

[0033] The present application also provides a method for processing multi-threaded converged time series data, which is implemented based on a multi-threaded converged time series data processing system, wherein the method is applied to a data cache component end, and the method includes the following steps:

[0034] Establish a cache queue;

[0035] The cache queues are matched to the rule groups.

[0036] The benefits of the present invention include:

[0037] 1. A multi-threaded time series data processing system and method are provided to avoid the problems of low performance and excessive storage log volume caused by using transactions or multiple attempts to read, and to ensure the accuracy of the final data.

[0038] 2. At the same time, compared with the existing technology, the technical solution provided by the present application does not perform log storage, but only performs log caching, thus saving a large amount of disk space; there is no need to verify the accuracy of data in the later retrieval method, and only the data required by the rules is retained within a certain period of time, thus eliminating the performance of later retrieval and storage; the timer can also be adjusted according to actual needs, ensuring data accuracy and shortening the alarm delay within a certain range; the calculations in the method are basically based on the number of threads, and the time complexity is O(1), and the performance of the later retrieval method is much higher; data correspondence processing is performed between threads throughout the entire process, and within a certain range, the performance can be linearly improved by simply increasing the number of threads. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram corresponding to the rule matching thread group and the data aggregation thread group;

[0040] Figure 2 Indicates intention for the chain;

[0041] Figure 3 A schematic diagram of all thread linked list headers;

[0042] Figure 4 Another schematic diagram of all thread linked list headers;

[0043] Figure 5 A schematic diagram of a cache queue. DETAILED DESCRIPTION

[0044] The following describes the implementation of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] It should be noted that the illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0046] A multi-threaded time series data processing system includes a rule matching component and a data cache component;

[0047] The rule matching component is used to connect to network traffic and extract protocol data in the traffic based on set rules;

[0048] The data cache component is used to cache the ordered protocol data that has been aggregated, and provide a first-in-first-out reading interface.

[0049] At the same time, in specific implementation scenarios, such as Figure 1 As shown, the rule matching component includes a rule matching thread group and a data aggregation thread group;

[0050] The rule matching thread group includes multiple rules, filters target data from the received network traffic data based on the set rules, and pushes the target data to the data aggregation thread group;

[0051] The data aggregation thread group has the same number of threads as the rule matching thread group, and the threads in the data aggregation thread group correspond one-to-one to the threads in the rule matching thread group.

[0052] It should be pointed out that the rule matching thread group only extracts the data required for subsequent business to avoid useless data increasing the memory burden. Figure 1 The data of S1 in the system will only be pushed to D1 due to the one-to-one correspondence, and the same applies to all subsequent threads. The number of thread groups in the system is determined at the beginning of the operation based on the hardware configuration. Since the received network traffic data is initially ordered, in the multi-threaded scenario, the single-threaded data is achieved in an ordered manner.

[0053] Furthermore, the data in each thread in the data aggregation thread group is stored through a linked list, and the content of each data is aggregated in the linked list according to the regular time granularity. After the data aggregation thread group receives the target data pushed by the rule matching thread group, it will pre-insert the data in a thread. If the data at the tail of the linked list in the thread is smaller than the pre-inserted data, new tail node data will be added. If the pre-inserted data is consistent with the data at the tail of the linked list, the corresponding data will be aggregated.

[0054] That is Figure 2 As shown in the figure, a linked list in a thread has received data of 13, 14, and 15. Each time new data is received, it will only be stored at the end of the linked list. If the data at the tail is smaller than the pre-inserted data, new tail node data needs to be added. If the data is consistent with the tail, the corresponding data is aggregated. This ensures that the aggregated data of each granularity time will not be divided into multiple parts.

[0055] Furthermore, a timer is set in the data aggregation thread group, and in each timer cycle, the data with the smallest time in the linked list header data of all data aggregation threads is taken. If there are multiple identical times, the first one is taken. While judging the minimum value, each linked list header data equal to the minimum value is marked. When the judgment is completed, all minimum values ​​are disconnected and pushed to the data cache component.

[0056] In the specific implementation scenario, one minute is selected as a timer cycle for example. The system time takes the smallest time data in the linked list header data of all data aggregation threads every minute. If there are multiple identical times, the first one is taken. While judging the minimum value, mark each linked list header data that is equal to the minimum value. When the judgment is completed, all minimum values ​​are disconnected and pushed to the cache queue.

[0057] like Figure 3 As shown, all thread list heads have a minimum time of 12, and subsequent data are also larger data. In this case, the timer will push all list heads to the cache queue.

[0058] like Figure 4 As shown in the figure, the linked list heads of some threads have processed the data at time 13. When the timer is triggered, only the data at time 12 will be pushed. Since the number of threads is fixed, the computational complexity here is O (1).

[0059] The purpose of this design is to use a small amount of calculation in exchange for the relative order of all rule data. After this step, the cache queue is in order.

[0060] Furthermore, the data cache component includes multiple cache queues. The data cache component provides an interface for applying for queues. It applies for and connects to the rule matching component during initialization. The data cache component saves data in memory. When new data is loaded into the cache, it is added to the end of the queue. When data needs to be read from the cache, the data is taken from the head of the queue. Each of the cache queues matches the rules in the rule matching thread group, and the same batch of rules will only be pushed to the same cache queue.

[0061] In specific usage scenarios, the data cache component provides an interface for applying for a queue, which is applied for and connected to the rule matching component during initialization. The main function is to implement first-in-first-out queue operations and cache data in memory. When new data is loaded into the cache, it is added to the end of the queue. When data needs to be read from the cache, the data is taken from the head of the queue. If the cache is full and new data needs to be loaded, the data at the head of the queue (that is, the data that entered the cache the earliest) will be removed to make room for the new data.

[0062] At the same time, in order to speed up the consumption of the cache queue, multiple cache queues are used to receive data. Multiple cache queues are created according to the number of rules. The rules pushed to the cache queue are grouped by rules, and the same batch of rules will only be pushed to the same cache queue. This ensures that subsequent analysis does not need to be read and aggregated from multiple queues. At this point, the data of a rule is aggregated. At this point, the data of the same rule is time-continuous in the cache queue, which can greatly facilitate subsequent processing.

[0063] like Figure 5 As shown, the first column represents the rules, the second column represents the time, different rules may appear alternately, but the same rules are continuous in the cache queue, and r1 and r2 represent different rules, 12 and 13 represent the time of the linked list head.

[0064] The purpose of this design is that the cache queue provides a first-in-first-out interface to specific business components, so that business components can obtain regular time series data in a streaming manner, which is very convenient for processing alarms or statistical information.

[0065] In this embodiment, a method for processing multi-threaded aggregated time series data is provided, which is applied to a rule matching thread group end, and the method includes the following steps:

[0066] Receive network traffic data;

[0067] Filter target data from network traffic data based on set rules;

[0068] Push the target data to the data aggregation thread group.

[0069] At the same time, in this embodiment, a method for processing multi-threaded aggregated time series data is provided, which is applied to a data aggregation thread group end, and the method includes the following steps:

[0070] Create a linked list and aggregate the content of each piece of data into the linked list according to the time granularity of the set rules;

[0071] Receive the target data pushed by the rule-matching thread group and pre-insert it into the linked list;

[0072] Within a timer cycle, the data with the smallest time in the linked list header data of all data aggregation threads is disconnected and pushed to the data cache component.

[0073] At the same time, in this embodiment, a multi-threaded aggregation time series data processing method is provided, which is applied to the data cache component end, and the method includes the following steps:

[0074] Establish a cache queue;

[0075] The cache queues are matched to the rule groups.

[0076] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-threaded time series data processing system, characterized in that: Includes rule matching components and data caching components; The rule matching component is used to connect to network traffic and extract protocol data in the traffic based on set rules; The data cache component is used to cache the ordered protocol data that has been aggregated, and provide a first-in-first-out reading interface.

2. According to claim 1, a multi-threaded time series data processing system is characterized in that: The rule matching component includes a rule matching thread group and a data aggregation thread group; The rule matching thread group includes multiple rules, filters target data from the received network traffic data based on the set rules, and pushes the target data to the data aggregation thread group; The data aggregation thread group has the same number of threads as the rule matching thread group, and the threads in the data aggregation thread group correspond one-to-one to the threads in the rule matching thread group.

3. A multi-threaded time series data processing system according to claim 2, characterized in that: The data in each thread in the data aggregation thread group is stored in a linked list, and the content of each data is aggregated in the linked list according to a regular time granularity.

4. The multi-threaded time series data processing system according to claim 3, characterized in that: After the data aggregation thread group receives the target data pushed by the rule matching thread group, it will pre-insert the data in a thread. If the data at the tail of the linked list in the thread is smaller than the pre-inserted data, new tail node data will be added. If the pre-inserted data is consistent with the data at the tail of the linked list, the corresponding data will be aggregated.

5. The multi-threaded time series data processing system according to claim 3, characterized in that: A timer is provided in the data aggregation thread group, and within each timer cycle, the data with the smallest time in the linked list header data of all data aggregation threads is taken. If there are multiple identical times, the first one is taken. While judging the minimum value, each linked list header data equal to the minimum value is marked. When the judgment is completed, all minimum values ​​are disconnected and pushed to the data cache component.

6. A multi-threaded time series data processing system according to claim 5, characterized in that: The data cache component includes multiple cache queues. The data cache component provides an interface for applying for queues. It applies for and connects to the rule matching component during initialization. The data cache component saves data in memory. When new data is loaded into the cache, it is added to the end of the queue. When data needs to be read from the cache, the data is taken from the head of the queue.

7. The multi-threaded time series data processing system according to claim 6, characterized in that: Each of the cache queues matches the rules in the rule matching thread group, and the same batch of rules will only be pushed to the same cache queue.

8. A method for processing multi-threaded converged time series data, implemented based on a multi-threaded converged time series data processing system in claim 2, characterized in that: Applied to the rule matching thread group end, the method includes the following steps: Receive network traffic data; Filter target data from network traffic data based on set rules; Push the target data to the data aggregation thread group.

9. A method for processing multi-threaded converged time series data, implemented based on a multi-threaded converged time series data processing system in claim 5, characterized in that: Applied to the data aggregation thread group end, the method includes the following steps: Create a linked list and aggregate the content of each piece of data into the linked list according to the time granularity of the set rules; Receive the target data pushed by the rule-matching thread group and pre-insert it into the linked list; Within a timer cycle, the data with the smallest time in the linked list header data of all data aggregation threads is disconnected and pushed to the data cache component.

10. A method for processing multi-threaded converged time series data, implemented based on a multi-threaded converged time series data processing system in claim 7, characterized in that: Applied to the data cache component side, the method includes the following steps: Establish a cache queue; The cache queues are matched to the rule groups.

Images