A Scheduling Method for Intrusion Detection System Based on Task Division

Abstract

The present invention belongs to the field of task adjustment, and specifically relates to an intrusion detection system scheduling method based on task division. The present invention obtains the load information of the processing core, obtains the processing core with the highest load and the processing core with the lowest load according to the load information; obtains the number of flows in the processing core with the highest load and the processing core with the lowest load and the load distribution of all flows; obtains the load difference between the processing core with the highest load and the processing core with the lowest load according to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load; judges whether there is an elephant flow in the processing core according to the number of flows in the processing core with the highest load and the processing core with the lowest load and the load difference of the processing core; if there is an elephant flow, the elephant flow is migrated after task division; if there is no elephant flow, the flow in the processing core with the highest load is migrated to the processing core with the lowest load. The migration flow between the processing cores can be balanced, and the load of the processing core is optimized.

Description

Technical Field

[0001] The invention belongs to the field of processing core task adjustment, and in particular relates to an intrusion detection system scheduling method based on task division. Background Art

[0002] In an IDS intrusion detection system, multiple redundant detection processing units are usually deployed to cope with dynamically changing input traffic. These processing units together form a system with elastic scalability to ensure that detection tasks can still be performed efficiently during traffic peaks. The IDS intrusion detection system performs a series of detection tasks on network data packets to identify possible intrusion risks. Different detection tasks belong to different network functions, which are arranged in a sequence of execution to form a set of service function chains (SFC). This type of IDS task is not sensitive to latency and real-time performance, and does not require high energy saving. It pays more attention to the utilization and stability of the detection system. If there is no single node failure or exceeds its load processing capacity, optimizing its task execution scheduling capability to improve system stability becomes an important direction.

[0003] For the task scheduling of network traffic, the most common way to improve utilization is to execute multiple processing units in parallel. Each processing unit is responsible for the detection task of a part of the traffic, which can significantly improve the overall processing capacity of the system. When dealing with excessive load, it is usually chosen to migrate part of the traffic to other processing units to ensure that the overall system load is relatively stable.

[0004] IDS detection is mainly used to monitor network traffic and system activities to identify possible malicious behavior or abnormal activities. Its task processing is mainly computationally intensive, such as packet capture, packet loading and unloading, protocol analysis, pattern matching, statistical analysis, behavioral analysis, contextual analysis, and alarm and response. This type of computing content involves data parsing, aggregation, sorting, and filtering, and has the characteristics of complex execution logic, large amounts of real-time data processing, and complex associations, comparisons, and reasoning, which requires huge computing resources.

[0005] Due to the complexity of IDS tasks, IDS systems require powerful computing power to support their normal operation. With the continuous increase in network traffic and attack methods, the demand for computing resources is also increasing, posing a huge challenge to the system scheduling capability. Increasing fine-grained task scheduling methods has become an important direction for improving the overall processing capability of the system.

[0006] However, scheduling data packets across processing units in a distributed multi-core system is a complex task, especially in scenarios where IDS intrusion detection systems need to process a large number of data packets in real time. Data packet processing tasks are divided into multiple stages, each of which involves different network functions. When tasks need to cross processing nodes between different stages, the consistency of task states must be ensured, such as using global flow tables, lock-free synchronization, and heartbeat mechanisms to ensure that state loss does not occur under the premise of reducing competition. During load balancing scheduling, data packets in the same task stage need to complete task migration between new and old nodes, involving synchronization of flow states and intermediate results, and network latency and data consistency need to be considered, such as only implementing incremental state migration and presetting migration buffers. Using centralized task scheduling and using centralized flow tables can speed up the collection of the entire cluster state information and make scheduling decisions based on the current load situation, but centralized scheduling may cause scheduling delays and single point failures, and centralized flow tables may also become bottlenecks in large-scale clusters. Dynamic task division and scheduling strategies can cope with real-time adjustments to various types of traffic, but in distributed clusters, how to dynamically adjust task division and ensure load balancing is still a difficult problem. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of scheduling delay and single point failure caused by adopting centralized energy regulation, and to provide an intrusion detection system scheduling method based on task division.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] The present invention provides an intrusion detection system scheduling method based on task division, comprising the following steps:

[0010] Acquire load information of the processing cores, and obtain a processing core with the highest load and a processing core with the lowest load according to the load information;

[0011] Obtain the number of flows in the processing core with the highest load and the processing core with the lowest load, as well as the load distribution of all flows;

[0012] According to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, the load difference between the processing core with the highest load and the processing core with the lowest load is obtained;

[0013] According to the number of flows in the processing core with the highest load and the processing core with the lowest load and the load difference of the processing cores, it is determined whether there is an elephant flow in the processing core;

[0014] If there is an elephant flow, the elephant flow will be divided into tasks and then migrated;

[0015] If there is no elephant flow, the flow in the processing core with the highest load is migrated to the processing core with the lowest load.

[0016] A further improvement of the present invention is that the specific method for obtaining the load information of the processing core is as follows:

[0017] The number of CPU cycles for processing data packets and the total number of CPU cycles of the target processing core within the pre-examination time period are obtained, and the load information of the processing core is obtained according to the ratio of the number of CPU cycles for processing data packets to the total number of CPU cycles.

[0018] A further improvement of the present invention is that, according to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, a specific method for obtaining the load difference between the processing core with the highest load and the processing core with the lowest load is as follows:

[0019] According to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, corresponding weights are set for the loads of all flows; wherein, when setting the corresponding weights for the loads of all flows, weights are assigned according to the processing complexity of the flows, and the higher the processing complexity of the flows, the greater the assigned weight;

[0020] According to the load weight values ​​of all flows in the processing core with the highest load and the processing core with the lowest load, the load weight of the processing core with the highest load and the load weight of the processing core with the lowest load are obtained;

[0021] According to the difference between the load weight of the processing core with the highest load and the load weight of the processing core with the lowest load, the load difference between the processing core with the highest load and the processing core with the lowest load is obtained.

[0022] A further improvement of the present invention is that the number of flows in the processing core with the highest load is less than a preset value of the flow number, and the load difference between the processing cores in the processing core with the highest load and the processing core with the lowest load is greater than a preset value of the load difference.

[0023] A further improvement of the present invention is that, if there is an elephant flow, the specific method of migrating the elephant flow after task division is as follows:

[0024] If there is an elephant flow, the number of flows in the processing core with the highest load is divided to obtain the data packets to be migrated; wherein, during the division process, when the processing core with the highest load and the processing core with the lowest load share different processing stages of a flow, the flow is preferentially migrated; when the processing core with the highest load and the processing core with the lowest load do not share different processing stages of a flow, the flow whose adjacent processing stages after migration can be jointly in the processing core with the lowest load is preferentially migrated, and the flow whose adjacent processing stages after migration can not be simultaneously in the processing core with the lowest load is preferentially migrated;

[0025] The processing core with the highest load distributes the divided data packets to be migrated to the processing core with the lowest load.

[0026] A further improvement of the present invention is that when the processing core with the highest load and the processing core with the lowest load do not share different processing stages of a flow, the processing core with the highest load and the processing core with the lowest load process the received flow, and after the processing is completed, send the processed data to the corresponding processing core.

[0027] A further improvement of the present invention is that when the processed data is sent to the corresponding processing core, if the processed data needs to be merged with the data of the previous stage, it is determined whether the data of the previous stage has been sent down. If it has been sent down, the data is merged; if it has not been sent down, the processed data is cached and waits for the data of the previous stage to be sent down.

[0028] A further improvement of the present invention is that when the processed data is sent to the corresponding processing core, if the processed data needs to be merged with the next stage, it is determined whether the data of the next stage has been processed. If it has been processed, the data is sent down and merged with the processed data of the next stage; if it has not been processed, the processed data is cached and waits for the data of the previous stage to be sent.

[0029] A further improvement of the present invention is that, if there is no elephant flow, the specific method of migrating the flow in the processing core with the highest load to the processing core with the lowest load is as follows:

[0030] If there is no elephant flow, the processing core with the highest load marks the flow packets that need to be migrated;

[0031] The processing core with the highest load distributes the data packets that need to be migrated to the corresponding queue of the processing core with the lowest load;

[0032] After receiving the complete data packet, the processing core with the lowest load starts processing the received complete data packet.

[0033] A further improvement of the present invention is that when the processing core with the highest load distributes the data packets that need to be migrated to the corresponding queue of the processing core with the lowest load, the data packets that are not completely received in the processing core with the lowest load are temporarily stored in the migration buffer queue, and the processing core with the lowest load processes other completely received data packets;

[0034] The receiving status of the data packet in the processing core with the lowest load is polled periodically. If the data packet is received completely, the processing of the received complete data packet begins.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention avoids the drawback of over-reliance on centralized scheduling by actively monitoring and analyzing the load information of the processing core and adjusting the traffic distribution according to the load difference. This decentralized scheduling method can quickly respond to load changes, thereby reducing the delay that may be caused by centralized scheduling. The present invention avoids all traffic being concentrated on a few processing cores by judging whether there is an elephant flow and migrating it after task division, thereby reducing the risk of single point failure. If a processing core fails, the task can be migrated to other processing cores through the scheduling strategy to ensure the high availability and fault tolerance of the system. The present invention optimizes the load balancing of the processing core by migrating traffic between the processing cores with the highest and lowest loads. This strategy can ensure that the resources of each processing core are fully utilized, avoid overloading of some processing cores or idle resources, thereby improving the overall processing efficiency of the system. The present invention divides and migrates tasks for elephant flows, which can effectively reduce the pressure of a certain processing core carrying too much traffic, prevent a single processing core from becoming a bottleneck due to excessive traffic, and improve the system's processing capacity and response speed for large traffic. In summary, the present invention can optimize the performance of the intrusion detection system through load scheduling and traffic migration, improve the stability, efficiency and scalability of the system, and overcome the delay and single point failure problems that may be caused by traditional centralized scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the present invention;

[0038] Figure 2 A schematic diagram of the network function task partitioning and scheduling of the present invention. DETAILED DESCRIPTION

[0039] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0040] See also Figure 1 , an intrusion detection system scheduling method based on task division, comprising the following steps:

[0041] S1, obtaining load information of the processing cores, and obtaining a processing core with the highest load and a processing core with the lowest load according to the load information.

[0042] S2, obtains the number of flows in the processing core with the highest load and the processing core with the lowest load, as well as the load distribution of all flows.

[0043] S3, according to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, obtain the load difference between the processing core with the highest load and the processing core with the lowest load.

[0044] S4, judging whether there is an elephant flow in the processing core according to the number of flows in the processing core with the highest load and the processing core with the lowest load and the load difference of the processing cores.

[0045] S5: If there is an elephant flow, the elephant flow will be migrated after task division; if there is no elephant flow, the flow in the processing core with the highest load will be migrated to the processing core with the lowest load.

[0046] The present invention dynamically divides and migrates tasks between the highest-load and lowest-load processing cores based on load information, so that scheduling decisions can respond more quickly to changes in system load, avoiding delays caused by centralized scheduling. When the load changes, traffic and tasks can be migrated to the appropriate processing core in a timely manner, thereby improving the overall scheduling response speed.

[0047] Embodiment 1:

[0048] The specific method for obtaining the load information of the processing core in S1 is as follows:

[0049] The number of CPU cycles for processing data packets and the total number of CPU cycles of the target processing core within the pre-examination time period are obtained, and the load information of the processing core is obtained according to the ratio of the number of CPU cycles for processing data packets to the total number of CPU cycles.

[0050] This embodiment can comprehensively consider indicators such as flow rate, computational complexity of processing tasks, and processing core processing capabilities, and accurately reflect the load differences between different cores.

[0051] Embodiment 2:

[0052] In S3, based on the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, the specific method for obtaining the load difference between the processing core with the highest load and the processing core with the lowest load is as follows:

[0053] Step one, according to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, set corresponding weights for the loads of all flows; wherein, when setting corresponding weights for the loads of all flows, weights are assigned according to the complexity of flow processing, and the higher the complexity of flow processing, the greater the assigned weight.

[0054] Step 2: According to the load weight values ​​of all flows in the processing core with the highest load and the processing core with the lowest load, the load weight of the processing core with the highest load and the load weight of the processing core with the lowest load are obtained.

[0055] Step three, according to the difference between the load weight of the processing core with the highest load and the load weight of the processing core with the lowest load, the load difference between the processing core with the highest load and the processing core with the lowest load is obtained.

[0056] The elephant flow is when the number of flows in the processing core with the highest load is less than the preset value of the flow number, and the load difference between the processing core with the highest load and the processing core with the lowest load is greater than the preset value of the load difference. That is, after the number of flows in the processing core with the highest load is divided, there is still a large load difference. If there is a single elephant flow, the other flows are all mouse flows.

[0057] Embodiment 3:

[0058] In S5, if there is an elephant flow, the specific method of migrating the elephant flow after task division is as follows:

[0059] Step 1: If there is an elephant flow, the number of flows in the processing core with the highest load is divided to obtain the data packets to be migrated; wherein, during the division process, when the processing core with the highest load and the processing core with the lowest load share different processing stages of a flow, it is preferred to migrate the said flow; when the processing core with the highest load and the processing core with the lowest load do not share different processing stages of a flow, it is preferred to migrate the flows in the processing core with the lowest load in adjacent processing stages after migration, and secondly, it is preferred to migrate the flows in the processing core with the lowest load in adjacent processing stages after migration that cannot be simultaneously in the processing core with the lowest load.

[0060] Step 2: The processing core with the highest load distributes the divided data packets to be migrated to the processing core with the lowest load.

[0061] The logic of task division gives priority to decision-making schemes that do not increase or decrease the number of task divisions, reducing the overhead of transmission between processing cores. When the processing core with the highest load and the processing core with the lowest load do not share different processing stages of a flow, the processing core with the highest load and the processing core with the lowest load process the received flow. After processing, the processed data is sent to the corresponding processing core.

[0062] When sending the processed data to the corresponding processing core, if the processed data needs to be merged with the previous stage, and the execution stage after migration is merged with the previous stage to the same processing core, determine whether the data of the previous stage has been sent down. If it has been sent down, merge the data; if it has not been sent down, cache the processed data in the migration buffer queue and wait for the data of the previous stage to be sent down. After the data is sent down, process all the data packets in the migration buffer queue first, and then process other received data packets to prevent the migration flows from different sources from being out of order. After clearing the migration buffer queue, you can reset the migration status and wait for the subsequent arriving data packets to be merged and processed.

[0063] When the processed data is sent to the corresponding processing core, if the processed data needs to be merged with the next stage, it is determined whether the data of the next stage has been processed. If it has been processed, the data is sent down and merged with the processed data of the next stage. If it has not been processed, the processed data is cached and waits for the data of the previous stage to be sent down. After the data is sent down and merged, the migration buffer queue is cleared, and the migration status is reset, waiting for the subsequent arrival of migration data to be merged.

[0064] For the split after migration, the processing core of the previous stage is the same as the processing core of the migrated processing stage, and there is a cross-processing core packet transfer after migration. After processing the migration flow packet, this task execution core only executes the post-migration processing part, and changes the corresponding metadata in each packet in time when sending, and ensures that all the previously processed pre-split packets are sent out, so that the migration synchronization can be completely completed to prevent the subsequent packets from being out of order due to queuing and sending queue delays.

[0065] Embodiment 4:

[0066] If there is no elephant flow in S5, the specific method of migrating the traffic from the processing core with the highest load to the processing core with the lowest load is as follows:

[0067] Step 1: If there is no elephant flow, the processing core with the highest load marks the flow packets that need to be migrated.

[0068] Step 2: The processing core with the highest load distributes the data packets that need to be migrated to the corresponding queue of the processing core with the lowest load.

[0069] Step 3: After receiving the complete data packet, the processing core with the lowest load starts to process the received complete data packet.

[0070] When the processing core with the highest load distributes the data packets that need to be migrated to the corresponding queue of the processing core with the lowest load, the data packets that are not completely received in the processing core with the lowest load are temporarily stored in the migration buffer queue, and the processing core with the lowest load processes other completely received data packets;

[0071] The receiving status of the data packet in the processing core with the lowest load is polled periodically. If the data packet is received completely, the processing of the received complete data packet begins.

[0072] Since the migration operation only changes the destination of the migrated data packet, and the processing load of the destination core is lower than that of the original processing core, the migration cache operation will not cause a large cache delay, but will enable the overall migration flow to be processed faster.

[0073] See also Figure 2The present invention adopts a design that decouples flow state, network function processing tasks, and execution cores, allowing any network function of any flow to be executed on any processing core. Each level of network function has a separate sending queue, which sends to the multi-producer single-consumer queue of the next level of network function.

[0074] The present invention uses isomorphic processing units and code execution logic to separate network function execution from scheduling, changing from using centralized distribution of data packets to achieve load balancing to all cores being able to dynamically schedule a certain stream of network function pending logic to other cores for processing at each network function execution stage, realizing the execution of any network function logic of any stream on any core, and fine-grained dynamic adjustment of inter-core loads. Software queues are used for forwarding at each level of network function processing, so the control plane can change the flow state to instruct the producer of the diversion queue to change the destination of the data packet after statistically analyzing the global traffic distribution and the global processing core load, thereby achieving dynamic task division.

[0075] The present invention supports the dynamic arrangement of network functions on the processing unit, and realizes the splitting, merging and migration of network functions according to the degree of load balancing based on the global scheduling flow table. For the traffic scenarios of elephant flow and hash polarization, a fine-grained scheduling space is provided for dynamic load balancing. The present invention divides and fine-grainedly schedules the network function execution chain of the data packet, and implements a multi-stage load balancing system to verify the performance of phased processing of network functions. Under the same unbalanced input traffic, after realizing multi-stage task division and diversion scheduling of network functions, the load deviation is reduced from 27% to 9%. After realizing multi-stage task division and diversion scheduling of network functions on the dynamic load balancing system, the load deviation is reduced from 17% to 2%.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for scheduling an intrusion detection system based on task division, characterized in that: The following steps are involved: Acquire load information of the processing cores, and obtain a processing core with the highest load and a processing core with the lowest load according to the load information; Obtain the number of flows in the processing core with the highest load and the processing core with the lowest load, as well as the load distribution of all flows; According to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, the load difference between the processing core with the highest load and the processing core with the lowest load is obtained; According to the number of flows in the processing core with the highest load and the processing core with the lowest load and the load difference of the processing cores, it is determined whether there is an elephant flow in the processing core; If there is an elephant flow, divide the elephant flow into tasks and then migrate it. The specific method is as follows: If there is an elephant flow, the number of flows in the processing core with the highest load is divided to obtain the data packets to be migrated; wherein, during the division process, when the processing core with the highest load and the processing core with the lowest load share different processing stages of a flow, the flow is preferentially migrated; when the processing core with the highest load and the processing core with the lowest load do not share different processing stages of a flow, the flow whose adjacent processing stages after migration can be jointly in the processing core with the lowest load is preferentially migrated, and the flow whose adjacent processing stages after migration can not be simultaneously in the processing core with the lowest load is preferentially migrated; The processing core with the highest load distributes the divided data packets to be migrated to the processing core with the lowest load; If there is no elephant flow, the flow in the processing core with the highest load is migrated to the processing core with the lowest load.

2. The method for scheduling an intrusion detection system based on task division according to claim 1, characterized in that: The specific method for obtaining the load information of the processing core is as follows: The number of CPU cycles for processing data packets and the total number of CPU cycles of the target processing core within the pre-examination time period are obtained, and the load information of the processing core is obtained according to the ratio of the number of CPU cycles for processing data packets to the total number of CPU cycles.

3. The method for scheduling an intrusion detection system based on task division according to claim 1, characterized in that: According to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, a specific method for obtaining the load difference between the processing core with the highest load and the processing core with the lowest load is as follows: According to the load distribution of all flows in the processing core with the highest load and the processing core with the lowest load, corresponding weights are set for the loads of all flows; wherein, when setting the corresponding weights for the loads of all flows, weights are assigned according to the processing complexity of the flows, and the higher the processing complexity of the flows, the greater the assigned weight; According to the load weight values ​​of all flows in the processing core with the highest load and the processing core with the lowest load, the load weight of the processing core with the highest load and the load weight of the processing core with the lowest load are obtained; According to the difference between the load weight of the processing core with the highest load and the load weight of the processing core with the lowest load, the load difference between the processing core with the highest load and the processing core with the lowest load is obtained.

4. The method for scheduling an intrusion detection system based on task division according to claim 1, characterized in that: The elephant flow is when the number of flows in the processing core with the highest load is less than a preset value of the flow number, and the load difference between the processing cores in the processing core with the highest load and the processing core with the lowest load is greater than a preset value of the load difference.

5. The method for scheduling an intrusion detection system based on task division according to claim 1, characterized in that: When the processing core with the highest load and the processing core with the lowest load do not share different processing stages of a flow, the processing core with the highest load and the processing core with the lowest load process the received flow, and after the processing is completed, send the processed data to the corresponding processing core.

6. The method for scheduling an intrusion detection system based on task division according to claim 5, characterized in that: When sending the processed data to the corresponding processing core, if the processed data needs to be merged with the previous stage, determine whether the data of the previous stage has been sent down. If it has been sent down, merge the data; if it has not been sent down, cache the processed data and wait for the data of the previous stage to be sent down.

7. The method for scheduling an intrusion detection system based on task division according to claim 5, characterized in that: When sending the processed data to the corresponding processing core, if the processed data needs to be merged with the next stage, determine whether the data of the next stage has been processed. If it has been processed, send the data and merge it with the processed data of the next stage; if it has not been processed, cache the processed data and wait for the data of the previous stage to be sent.

8. The method for scheduling an intrusion detection system based on task division according to claim 1, characterized in that: If there is no elephant flow, the specific method of migrating the traffic from the processing core with the highest load to the processing core with the lowest load is as follows: If there is no elephant flow, the processing core with the highest load marks the flow packets that need to be migrated; The processing core with the highest load distributes the data packets that need to be migrated to the corresponding queue of the processing core with the lowest load; After receiving the complete data packet, the processing core with the lowest load starts processing the received complete data packet.

9. The method for scheduling an intrusion detection system based on task division according to claim 8, characterized in that: When the processing core with the highest load distributes the data packets that need to be migrated to the corresponding queue of the processing core with the lowest load, the data packets that are not completely received in the processing core with the lowest load are temporarily stored in the migration buffer queue, and the processing core with the lowest load processes other completely received data packets; The receiving status of the data packet in the processing core with the lowest load is polled periodically. If the data packet is received completely, the processing of the received complete data packet begins.

Images