Dynamic load balancing method for multiple parallel anomaly detection tasks

Abstract

The invention discloses a dynamic load balancing method for multiple parallel anomaly detection tasks. According to the dynamic load balancing method oriented to the multiple parallel anomaly detection tasks, aiming at the characteristics of large data volume, strong real-time performance, obvious periodicity and the like of anomaly detection, a solution comprising a multi-process shared cache pool, incremental load period trend prediction and task dynamic allocation is provided; therefore, an efficient and perfect load balancing mechanism is realized. The method can be divided into three parts: detecting a parallel task scheduling process framework under the constraint of a business ontology, wherein the parallel task scheduling process framework comprises normalized description of an anomaly detection task and a shared data exchange mechanism; a task load dynamic prediction mechanism based on an SARIMA method is used for predicting the abnormal detection task load trend in a short period; and based on a multi-task dynamic hot start scheduling mechanism of a KM algorithm, calculating the benefit of each sub-task allocated to each process on the basis of a prediction result of the SARIMA, so as to realize the optimal allocation of the tasks. The method can be used for scenes with large data volume or strong computing resource constraint such as big data telemetering and on-board anomaly detection, and more stable and reliable anomaly detection is realized.

Description

Technical Field

[0001] The invention relates to a dynamic load balancing method, in particular to a dynamic load balancing method oriented to multiple parallel anomaly detection tasks. Background Art

[0002] For complex systems, especially those with high precision and reliability requirements, high-real-time on-device anomaly detection functions are very important to ensure the normal execution of related functions. As the operating cycle of current products in the fields of aviation, aerospace, and shipbuilding continues to increase, the reliability requirements of products are becoming more and more stringent. High-real-time on-device anomaly detection has gradually become one of the important functions of product reliability. However, in special fields such as aviation, aerospace, and shipbuilding, current products also have development trends such as increasingly complex functional structures and high-frequency signal sampling. In the actual operating environment, there are business characteristics such as strong computing resource limitations and large real-time data throughput, which introduces new performance challenges for on-device or edge-side anomaly detection at the scheduling level.

[0003] Considering that the anomaly detection on each signal combination can be executed independently as a different task, the anomaly detection on the device is essentially a typical multi-task parallel scheduling problem. The existing solutions to the multi-task parallel scheduling problem mainly include two typical types: traditional passive parallel scheduling methods and scheduling methods based on static optimization algorithms.

[0004] Traditional passive parallel scheduling methods mainly add divide-and-conquer and fork-join ideas that adapt to parallel computing to basic scheduling rules such as round-robin scheduling, first come first served, and highest response ratio next, so as to achieve parallel scheduling. The advantage of this type of method is that the basic principle is simple, but the high degree of abstraction makes it mostly used in the underlying algorithm optimization scenario, and cannot be applied to the complex business of on-device anomaly detection, which has a dynamic process and obscure mechanism.

[0005] The scheduling method based on static optimization algorithm is mainly designed with operation and maintenance experience, and a priori modeling of task load and peak time is performed. Then, optimization algorithms including Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Simulated Annealing Algorithm and other optimization algorithms are used to statically allocate computing resources to tasks. This type of method can find the optimal solution in a given scenario, but it cannot effectively schedule the dynamic changes of detection task load in the actual complex equipment anomaly detection business scenario.

[0006] The above methods are all significantly passive and static in terms of task allocation. At the same time, they cannot adapt to the personalized problem of slow startup of some anomaly detection tasks, and cannot meet the high real-time and high throughput requirements of on-device anomaly detection. Therefore, a dynamic task scheduling method based on load prediction is of great significance for the effective implementation of on-device anomaly detection functions. The dynamic load balancing method for multi-parallel anomaly detection tasks can effectively improve the reasoning efficiency of complex system anomaly detection and ensure full security and reliability. Summary of the invention

[0007] In order to solve the problems of insufficient performance optimization and insufficient computing power utilization in the anomaly detection process, the present invention provides a dynamic load balancing method for multiple parallel anomaly detection tasks. The dynamic load balancing method for multiple parallel anomaly detection tasks is aimed at scenarios with strong optimization requirements such as limited performance or huge data volume. It evaluates the estimated occupied resources of the task and the available resources in the parallel pool for scheduling and allocation, thereby achieving dynamic load balancing.

[0008] The dynamic load balancing method is aimed at anomaly detection software and hardware systems with high real-time requirements, large data volumes, or strong computing power constraints, and can be used in a variety of scenarios such as airborne or vehicle-mounted real-time health monitoring, big data telemetry, etc.

[0009] The dynamic load balancing method first determines the process pool opened for the executed anomaly detection task and determines the key constraints that restrict the computing efficiency, such as CPU occupancy or memory occupancy. After that, the computing power demand evaluation based on the algorithm complexity and the periodic trend evaluation of the global load are performed for the subtasks generated in the anomaly detection process, and the optimal allocation scheme of the subtask in the process pool is obtained based on this. Finally, the information exchange and resource occupancy monitoring between processes are realized simultaneously through the database middleware, and then a perfect process interaction mechanism is established on the basis of load balancing.

[0010] The present invention is characterized in that:

[0011] (1) Based on the process shared cache pool framework, a full life cycle and data structure management mechanism for tasks in a highly parallel computing cluster is constructed, thereby realizing the standardized scheduling function of multiple heterogeneous computing tasks at the business level;

[0012] (2) Based on the incremental periodic trend prediction method, an efficient real-time evaluation technology for process computing resources is constructed, thereby realizing the dynamic allocation function of shared computing resources under the dynamic evolution characteristics of multi-task loads;

[0013] (3) Based on the KM algorithm, the problem of allocating subtasks among processes is converted into a bipartite graph optimal matching problem to achieve a multi-task dynamic hot start mobilization mechanism and better load balancing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of the shared process pool framework structure.

[0015] Figure 2 A schematic diagram of a hierarchical data management structure.

[0016] Figure 3 Schematic diagram of the method for assigning subtasks. DETAILED DESCRIPTION

[0017] A dynamic load balancing method for multiple parallel anomaly detection tasks provided by the present invention is described in detail below with reference to the accompanying drawings.

[0018] The present invention provides a dynamic load balancing method for multiple parallel anomaly detection tasks. The dynamic load balancing method is oriented to anomaly detection scenarios with strong real-time requirements, limited computing power or high performance optimization requirements, and provides a solution with both generality and scalability. The dynamic load balancing method can be divided into three main parts: a parallel task scheduling process framework under the constraints of the detection business ontology, a task load dynamic prediction mechanism based on the SARIMA method, and a multi-task dynamic mobilization mechanism based on the KM algorithm.

[0019] 1. Parallel task scheduling process framework under the constraints of detection business ontology

[0020] The dynamic load balancing method first realizes the standardized management of each anomaly detection subtask by establishing a parallel task scheduling framework. The shared process framework is mainly responsible for the core elements of the task execution process based on the ontological characteristics of the entire life cycle of the anomaly detection task, and at the same time constructs a process shared cache space for the hot start requirements of the anomaly detection task, alleviating the efficiency defects of the loading process of the anomaly detection model represented by the neural network.

[0021] The standardized description of the anomaly detection core elements mainly includes the algorithm name used in the specific anomaly detection task, the corresponding algorithm parameters, the signal name involved, the time interval of analysis, the abnormal result of reasoning, and auxiliary information for manual anomaly analysis and processing such as anomaly description and plan information. The shared process framework constructs a standardized anomaly detection instance based on the standardized description of the anomaly detection core elements.

[0022] In the planned anomaly detection instance generation mechanism, the shared process framework is as follows Figure 1As shown, the information exchange between processes is mainly realized through the shared cache data exchange mechanism. The shared cache data exchange mechanism is mainly realized through middleware represented by Redis. The results and variables of the internal computing tasks of the process are stored in the middleware. The middleware relies on memory implementation and can be accessed through the network to ensure full versatility and efficiency.

[0023] Specifically, if Figure 2 As shown, the process shared cache pool opens up the field domain AnomDetectTask, and stores the information in the form of a standardized dictionary or key-value pair. The information in the cache pool is mainly divided into two categories: signal data and public variables. The storage form of the signal data in the middleware follows the hierarchical relationship of signal-sub-entry-data. A single signal can contain a single or multiple sub-entries, and the sub-entries contain specific data. The shared variables in the middleware storage include both metadata and variable content. In addition to storing the content of the variable in the form of a dictionary under a single variable entry, meta information is also included when writing. The meta information includes the sub-process ID that generates the variable, the name of the measurement point involved, and the time interval analyzed. Based on the process shared cache pool, the computing resources of each process are managed in real time.

[0024] The dynamic load balancing method constructs a hierarchical data management structure of main program cache-middleware-database. The main program cache stores all existing subtask information, including subtask ID, the name of the measurement point data item involved, and the required information such as the algorithm type involved; the middleware stores commonly used measurement point data, common variables between processes, and resource usage information of each subprocess in the process pool to reduce the burden of large amounts of IO access; the data content included in the database mainly includes parameters of the anomaly detection algorithm model, anomaly detection evaluation results, and important logs. The update frequency of the database is much lower than that of the middleware, and it mainly serves the function of saving important data.

[0025] 2. Dynamic prediction mechanism of task load based on SARIMA method

[0026] In order to adapt to the unique periodic characteristics of anomaly detection tasks, the anomaly detection framework program monitors the computing resource usage of each process in the process pool in real time, and on this basis, realizes deep perception of task trend nature information through the task load dynamic prediction mechanism. Specifically, the task load dynamic prediction mechanism is implemented through the SARIMA (Seasonal Auto Regressive Integrated Moving Average Model) algorithm.

[0027] The SARIMA method includes two components: seasonality and non-seasonality. Both parts include three sub-items: sequence difference, moving average and autoregression. Among them, the sequence difference operation is performed on the most recent item in the sequence. The main function of this operation is to eliminate the trend or non-stationary properties in the sequence through d-order differences; autoregression realizes the capture of the autocorrelation of the difference sequence. This operation represents the autoregression term as a linear combination of the most recent p terms in the difference sequence; the moving average term captures the random fluctuations in the difference sequence. This operation performs a sliding average on the most recent q terms in the residual term sequence of the model fitting; seasonality and non-seasonality have the same three sub-operations, but the difference is that the non-seasonal part examines the latest max(p,q)+d terms of the target sequence before the current time point t, while the seasonal part examines the latest max(p,q)+d terms before ts. The final prediction result is the sum of all seasonal and non-seasonal sub-items.

[0028] 3. Multi-task dynamic hot start mobilization mechanism based on KM algorithm

[0029] In order to achieve a balanced load, it is necessary to determine the process to which each subtask is assigned based on the resource occupancy; the dynamic load balancing method achieves this operation through a multi-task dynamic mobilization method. The multi-task dynamic mobilization method mainly includes two stages: task allocation benefit estimation and task hot start decision, both of which are carried out at the beginning of the subtask, aiming to execute the task initialization process in advance and allocate a more suitable computing environment for different anomaly detection tasks.

[0030] The task allocation benefit estimation phase mainly combines the predicted short-term subtasks and load conditions with the current process resource occupancy information to generate a subtask-subprocess bipartite graph. For each subtask that is at the beginning or predicted to start, the complexity index is used to evaluate the computing resources that may be consumed during its execution. At the beginning of the subtask, the scheduler assigns the task to the appropriate process based on the total computing resource usage of anomaly detection. The benefit of a single subtask i assigned to process j is derived from the following formula:

[0031]

[0032] Where: task i represents the estimated resource requirements of task i, rest j represents the remaining resources of process j, and global represents the overall available resource scale for anomaly detection; Q ij represents the benefit of task i in the allocation of process j; task min With task max Represents the minimum and maximum resource requirements of a task. and Distribution represents the sufficiency benefit that a task brings to other processes and the availability benefit that a task brings within a process. Represents the coefficient of difference in task resource usage. The more significant the memory requirements between tasks, the more it is necessary to consider the impact of this allocation on subsequent task allocations when allocating tasks; the less significant it is, the more it is necessary to ensure that the task has sufficient computing space within the process. The benefit of each subtask allocated to each process is the weight of the corresponding edge on the bipartite graph. Let the set of subtasks be U, the set of processes be V, and the set of edges be E, then the entire bipartite graph can be described as G = {U, V, E}.

[0033] The task allocation decision stage uses the KM (Kuhn-Munkras) algorithm to schedule subtasks based on the obtained bipartite graph. Specifically, the goal of the algorithm is to find the match with the largest weight in the given bipartite graph G, so as to find the most suitable computing power node in a short-term perspective for the business that needs hot start. The subtask scheduling mechanism based on the KM algorithm is as follows Figure 3 As shown, it mainly includes three main steps:

[0034] (1) Screen the bipartite graph and find all the expected benefits below the benefit threshold Q th The tasks are blocked, and the remaining tasks are used as hot start subtasks to be executed and enter the next step of decision making;

[0035] (2) The KM algorithm sets a top label for the subtask node in the bipartite graph, whose value l(u) is the maximum weight of all the links of the node. The top label l(v) of the process node is set to zero.

[0036] (3) KM algorithm considers subgraph G ′ ={U,V,E′}, E′ is the set of all edges that satisfy l(u)+l(v)=e(u,v).

[0037] If in G ′ If all nodes can be matched, then the match is the best match. Otherwise, go to the next step.

[0038] (4) For all unmatched nodes u, v, adjust their top-mark values. Take δ = min(l(u) + l(v) -

[0039] e(u,v)), reduce the top label l(u) of the unmatched u by δ, and increase the top label l(v) of the unmatched v by δ. Then return to the previous step and loop until the optimal solution is found.

[0040] The multi-task dynamic mobilization method further combines the middleware to perform the hot start step of the subtask based on the matching results in advance, that is, to create subtasks on the matching computing nodes and complete the initialization task of the anomaly detection algorithm. As the short-term subtasks are completed, the multi-task dynamic mobilization method will make decisions based on the most recent matching results. If the algorithm still participates in the next stage of tasks, it will directly wait for the time threshold T th , waiting for the data to arrive. If the waiting time threshold is exceeded and the corresponding anomaly detection task demand still does not appear, the subtask is destroyed and the node computing resources are released; otherwise, reasoning is performed directly based on the signal and time period of the new anomaly detection task. If the algorithm does not participate in the next stage task, the subtask is directly destroyed and the node computing resources are released.

Claims

1. A dynamic load balancing method for multiple parallel anomaly detection tasks, characterized by: In view of the characteristics of telemetry or on-device anomaly detection tasks such as large data volume, strong real-time nature, and obvious periodicity, solutions including multi-process shared cache pool, incremental load cycle trend prediction and dynamic task allocation are provided to achieve an efficient and complete load balancing mechanism.

2. According to claim 1, the dynamic load balancing method for multiple parallel anomaly detection tasks is characterized by: A process shared cache framework is established to achieve standardized description of anomaly detection subtasks. For the information involved in anomaly detection tasks, a hierarchical data management structure is built through cache, middleware and database to ensure the full universality and efficiency of the inter-process interaction mechanism and IO access mechanism.

3. According to claim 1, the dynamic load balancing method for multiple parallel anomaly detection tasks is characterized by: Through the seasonal autoregressive moving average model algorithm, the predicted value of the anomaly detection task load sequence is expressed as the sum of the seasonal term and the non-seasonal term. The two terms each contain three sub-terms: sequence difference, moving average and autoregression. Through integration, the load prediction of the anomaly detection task is finally achieved.

4. According to claim 1, the dynamic load balancing method for multiple parallel anomaly detection tasks is characterized by: The problem of allocating anomaly detection subtasks to processes is reduced to the optimal matching problem of the task-process bipartite graph. The benefit of allocating each subtask to each process is calculated by the estimated resources occupied by the tasks and the available resources of the processes. The function of allocating subtasks to processes is realized through the KM algorithm, achieving the goal of multi-task load balancing.

Images