Batch task buffer scheduling method and device based on three-level hierarchical architecture

Abstract

This invention discloses a batch task buffer scheduling method and apparatus based on a three-level hierarchical architecture. The method is implemented based on a global scheduling layer (Master), a node management layer (Agent), and a computation execution layer (Engine). First, a data processing task set is acquired. The global scheduling layer processes the data processing task set to obtain several batch task packages, which are then distributed to the node management layer. Next, the node management layer calls the computation execution layer to compute and execute the data processing tasks of the several batch task packages, obtaining aggregated task processing status information, which is then reported to the global scheduling layer. Finally, the global scheduling layer determines whether all data processing tasks have been completed based on the aggregated task processing status information. If not, the data processing task set is updated and the scheduling is repeated until all data processing tasks are completed. This application can be applied to large-scale distributed systems, meeting the requirements for high throughput and low latency task scheduling.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and task scheduling technology, and in particular to a batch task buffer scheduling method and apparatus based on a three-level hierarchical architecture. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and the Internet of Things (IoT) technologies, distributed system scheduling technology is facing unprecedented technical challenges and opportunities. In the wave of digital transformation, enterprises' demand for computing resources is growing exponentially, especially in fields such as artificial intelligence, big data analytics, and scientific computing, where the performance requirements for task scheduling systems are increasingly stringent.

[0003] Traditional distributed scheduling technologies primarily employ a request-response communication model. While this model performs well in small to medium-sized applications, it reveals significant performance bottlenecks when handling large-scale concurrent tasks. Each task submission requires a complete network round trip, including request sending, server processing, and response return. This frequent network interaction severely limits system throughput.

[0004] In terms of technical architecture, existing scheduling systems are mainly divided into two modes: centralized and distributed. Centralized scheduling architectures, such as traditional job scheduling systems, are simple to implement and easy to manage, but they have the risk of single point of failure; if the master node fails, the entire system will be paralyzed. Distributed scheduling architectures, such as systems based on P2P or consensus algorithms, improve the system's fault tolerance, but face technical challenges such as complex task coordination and difficulty in maintaining consistency.

[0005] Of particular note is the application of high-throughput computing (HTC) scenarios, such as simulation of tens of millions of discrete events and large-scale parameter scanning. These applications typically need to process a massive number of short-duration tasks. Each task has a short execution time but a huge number of tasks, which places extremely high demands on the throughput and latency of the scheduling system. Traditional scheduling methods are difficult to meet these requirements, and there is an urgent need for a task scheduling method that can meet the requirements of high throughput and low latency. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a batch task buffer scheduling method and apparatus based on a three-level hierarchical architecture, which solves the technical problem that traditional task scheduling methods in the prior art cannot meet the requirements of high throughput and low latency.

[0007] To address the aforementioned technical problems, the first aspect of this invention discloses a batch task buffering and scheduling method based on a three-level hierarchical architecture. The method is implemented based on a global scheduling layer (Master), a node management layer (Agent), and a computation execution layer (Engine). The global scheduling layer (Master) includes a global scheduling center for global scheduling and task buffering. The node management layer (Agent) includes several node management units for implementing closed-loop control and task status aggregation at the node management level. The computation execution layer (Engine) includes several groups of computation execution units for computing and executing data processing tasks. The batch task buffer scheduling method includes: S1, Obtain a data processing task set; the data processing task set includes N data processing tasks, where N is a positive integer; S2, the global scheduling layer is used to process the data processing task set and the node management layer to obtain a valid data processing task set, M target node management units, several batch task packages, and several batch task packages and their mapping relationships with the M target node management units; the valid data processing task set includes R data processing tasks, where R is a positive integer and R≤N. S3, the global scheduling layer distributes the batch task packages to the M target node management units based on the mapping relationship between the batch task packages and the M target node management units; S4, the M target node management units use the computing execution layer to process the plurality of batch task packages to obtain a plurality of batch task processing status information packages; S5, the node management layer processes the plurality of batch task processing status information packets to obtain task processing status aggregate information; S6, the node management layer reports the task processing status aggregation information to the global scheduling layer; S7, the global scheduling layer determines, based on the task processing status aggregation information, whether all R data processing tasks in the legal data processing task set have been executed, and obtains a first judgment result; When the first judgment result is yes, the batch task buffer scheduling process ends; If the first judgment result is negative, execute S8; S8, the global scheduling layer updates the legitimate data processing task set based on the task processing status aggregation information to obtain the updated data processing task set, and determines the updated data processing task set as the data processing task set, and executes S2.

[0008] As an optional implementation, in the first aspect of the present invention, the step of using the global scheduling layer to process the data processing task set and the node management layer to obtain a valid data processing task set, M target node management units, several batch task packages, and a mapping relationship between the several batch task packages and the M target node management units includes: S21, the global scheduling layer uses the task verification model and parameter verification model to perform task verification and parameter verification on the data processing task set to obtain a valid data processing task set; the valid data processing task set includes R data processing tasks, where R is a positive integer and R≤N. The expression for the task verification model is: In the formula, The task verification value is the i-th data processing task in the data processing task set, and its value range is 0 or 1. This refers to the i-th data processing task in the data processing task set, where i = 1, 2, ..., N; For task identification integrity indicators, It is the globally unique identifier of the i-th data processing task in the data processing task set; As an indicator of the legality of the task status, This represents the initial state of the i-th data processing task in the data processing task set. As an indicator of the rationality of task resources, The amount of computing resources required for the i-th data processing task in the data processing task set; As an indicator of the effectiveness of task types, The business type of the i-th data processing task in the data processing task set; The expression for the parameter verification model is: In the formula, The parameter verification value is for the i-th data processing task in the data processing task set, and its value range is 0 or 1; The set of execution parameters for the i-th data processing task in the data processing task set; To implement the template validity index, The execution configuration template for the i-th data processing task in the data processing task set; This is an indicator of data path validity. The input data path for the i-th data processing task in the data processing task set. The output path of the i-th data processing task in the data processing task set; This is a CPU resource constraint indicator. The number of CPU cores required for the i-th data processing task in the data processing task set. The maximum CPU limit allowed by the system; As a memory resource constraint indicator, The memory size required for the i-th data processing task in the data processing task set. The maximum memory limit allowed by the system; To enforce timeout constraints, The estimated timeout for the execution of the i-th data processing task in the data processing task set. This is the minimum legal timeout time for the system; The expression for the set of legitimate data processing tasks is: In the formula, LegalTask ​​is the set of legal data processing tasks; R is the number of data processing tasks in the set of legal data processing tasks; S22, process the legal data processing task set and the node management layer to obtain M target node management units; S23, Based on the M target node management units, obtain the M buffers and the mapping relationship between the M buffers and the M target node management units; S24, using the M target node management units, the R data processing tasks of the legal data processing task set are allocated and processed to obtain M task queues and the mapping relationship between the M task queues and the M target node management units; each task queue includes several data processing tasks, and the sum of the queue lengths of the M task queues is R; S25, Based on the mapping relationship between the M task queues and the M target node management units and the mapping relationship between the M buffers and the M target node management units, the mapping relationship between the M task queues and the M buffers is obtained; S26, using the M buffers and the mapping relationship between the M task queues and the M buffers, perform dual-trigger batch buffering processing on the M task queues synchronously to obtain several batch task packages and several batch task packages and the mapping relationship between the M buffers. S27. Based on the mapping relationship between the batch task packages and M buffers and the mapping relationship between the M buffers and M target node management units, the mapping relationship between the batch task packages and M target node management units is obtained.

[0009] As an optional implementation, in a first aspect of the present invention, the processing of the legitimate data processing task set and the node management layer to obtain M target node management units includes: S221, Process the set of legal data processing tasks to obtain the required number M of node management units; S222, a heartbeat detection request is sent to the node management layer, and a heartbeat response message is received from the node management layer; S223, based on the preset node management unit health status determination rules and the heartbeat response message, the health status of several node management units in the node management layer is screened to obtain a candidate set of healthy node management units; the candidate set of healthy node management units includes L healthy node management units; the health status expression of the node management unit is: In the formula, This represents the health status of the j-th node management unit in the node management layer, with a value of 0 or 1. =1 indicates that the j-th node management unit of the node management layer is a healthy node. =0 indicates that the j-th node management unit of the node management layer is an unhealthy node; The network delay in the heartbeat response message of the j-th node management unit of the node management layer. The preset maximum allowable network latency; The service status code in the heartbeat response message of the j-th node management unit of the node management layer; The CPU real-time load rate in the heartbeat response message of the j-th node management unit of the node management layer. This is the preset maximum CPU load threshold; The memory usage rate is the real-time rate in the heartbeat response message of the j-th node management unit of the node management layer. This is the preset maximum memory usage threshold; The length of the current pending task queue of the j-th node management unit in the node management layer; S224, determine whether the number L of the health node management units is greater than or equal to the required number M of the node management units, and obtain a second judgment result; If the second judgment result is yes, execute S225; If the second judgment result is negative, execute S226; S225, Based on the candidate set of health node management units, M target node management units are obtained; S226, Create ML new node management units, and integrate the ML node management units and the L healthy node management units to obtain the M target node management units.

[0010] As an optional implementation, in the first aspect of the present invention, the step of using the M target node management units to allocate and process the R data processing tasks of the legitimate data processing task set to obtain M task queues and the mapping relationship between the M task queues and the M target node management units includes: S241, Based on the M target node management units, a target node management unit sequence list is obtained; the length of the target node management unit sequence list is M. S242, Based on the target node management unit sequence table, establish M corresponding empty task queues to obtain the mapping relationship between the M empty task queues and the M target node management units; S243, Based on the set of legal data processing tasks, a queue of data processing tasks to be assigned is obtained; the length of the queue of data processing tasks to be assigned is R; S244, based on the mapping relationship between the M empty task queues and the M target node management units, process the data processing task queue to be assigned and the M empty task queues to obtain the M task queues and the mapping relationship between the M task queues and the M target node management units.

[0011] As an optional implementation, in the first aspect of the present invention, the step of using the M buffers and the mapping relationship between the M task queues and the M buffers to synchronously perform dual-trigger batch buffering processing on the M task queues to obtain a plurality of batch task packages and a plurality of batch task packages and the mapping relationship between the M buffers includes: S261, Start the preset M timers to begin recording the task buffer duration of the corresponding buffers; S262, based on the mapping relationship between the M task queues and the M buffers, the data processing tasks of the M task queues are continuously input into the corresponding M buffers one by one according to a preset time step. S263, determine whether the number of data processing tasks in the M buffers has reached a preset number threshold, and obtain a third judgment result; If the third determination result is yes, execute S265; If the result of the third judgment is negative, execute S264; S264, determine whether the task buffer duration has reached the preset duration threshold, and obtain the fourth determination result; If the fourth determination result is yes, execute S265; If the result of the fourth judgment is negative, execute S262; S265, the data processing tasks in the buffer where the third judgment result is yes and / or the fourth judgment result is yes are packaged and output, and the buffer where the third judgment result is yes and / or the fourth judgment result is yes and the corresponding timer are reset; S266, repeat S262 to S265 until the length of all M task queues is 0, to obtain several batch task packages and several batch task packages and M buffer mapping relationships.

[0012] As an optional implementation, in the first aspect of the present invention, the M target node management units utilize the computation execution layer to process the plurality of batch task packages to obtain a plurality of batch task processing status information packages, including: S41, the M target node management units preprocess the batch task packages they receive to obtain M local task queues. S42, the M target node management units, based on the M local task queues, filter the computing execution layer to obtain M computing execution unit groups and the mapping relationship between the M target node management units and the M computing execution unit groups; each computing execution unit group includes several computing execution units; S43, the M target node management units, based on the mapping relationship between the M target node management units and the M computing execution unit groups, respectively call the corresponding computing execution unit groups to synchronously calculate and execute the M local task queues to obtain several task processing status information; S44, based on the intelligent status batch reporting mechanism, the processing status information of the several tasks is processed to obtain several batch task processing status information packages.

[0013] As an optional implementation, in the first aspect of the present invention, the step of processing the plurality of task processing status information based on the intelligent status batch reporting mechanism to obtain a plurality of batch task processing status information packets includes: S441, Preset task processing status information queue; the initial state of the task processing status information queue is an empty queue; S442, add the several task processing status information to the task processing status information queue one by one in the order they are generated in real time. S443, determine whether the length of the task processing status information queue is greater than or equal to a preset length threshold, and obtain the fifth judgment result; If the fifth judgment result is yes, execute S446; If the result of the fifth judgment is negative, execute S444; S444, Determine whether there is a batch mode disabled flag in the task processing status information queue, and obtain the sixth determination result; If the result of the sixth judgment is yes, execute S446; If the result of the sixth judgment is negative, execute S445; S445, determine whether there is a separate reporting identifier in the task processing status information queue, and obtain the seventh judgment result; If the result of the seventh judgment is yes, execute S446; If the seventh judgment result is negative, then it is determined whether all the task processing status information has been enqueued to obtain the eighth judgment result; When the eighth judgment result is yes, the task processing status information queue is packaged to obtain a batch task processing status information package, and then S5 is executed; If the result of the eighth judgment is negative, execute S446; S446, The task processing status information queue is packaged to obtain a batch task processing status information package. S447, the batch task processing status information packet is sent to the corresponding target node management unit, and the task processing status information queue is cleared. S448, update the task processing status information queue to obtain the updated task processing status information queue, and confirm the updated task processing status information queue as the task processing status information queue, then execute S442.

[0014] A second aspect of this invention discloses a batch task buffer scheduling device based on a three-level hierarchical architecture, the device comprising: processor; A memory coupled to the processor stores executable program code; The processor calls the executable program code stored in the memory to execute the batch task buffer scheduling method based on a three-level hierarchical architecture disclosed in the first aspect of the present invention.

[0015] A third aspect of the present invention discloses a computer-readable storage medium storing computer instructions, which, when invoked, are used to execute the batch task buffer scheduling method based on a three-level hierarchical architecture disclosed in the first aspect of the present invention.

[0016] The fourth aspect of the present invention discloses an information data processing terminal, characterized in that the information data processing terminal is used to implement the batch task buffer scheduling method based on a three-level hierarchical architecture disclosed in the first aspect of the present invention.

[0017] Compared with existing technologies, the embodiments of the present invention have the following beneficial effects: Utilizing a three-level hierarchical architecture including a global scheduling layer (Master), a node management layer (Agent), and a computation execution layer (Engine), data processing task sets are batch packaged, targeted, distributed, executed in a distributed manner, their status collected, aggregated in layers, and scheduled in a closed loop, achieving efficient task distribution, parallel execution, and unified status management; effectively solving the problems of frequent network interactions, high bandwidth consumption, low resource utilization, insufficient scheduling efficiency, untimely status monitoring, and easy task omissions in traditional distributed scheduling, significantly improving system throughput and resource utilization, ensuring a stable, complete, and omission-free scheduling process, and adapting to large-scale computing scenarios with high throughput and low latency requirements. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a batch task buffer scheduling method based on a three-level hierarchical architecture disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of a three-level hierarchical architecture disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of a batch task buffer scheduling device based on a three-level hierarchical architecture disclosed in an embodiment of the present invention; Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0024] It should be noted that since the method in this application embodiment is executed in a computer device, the processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information. It is understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the computer device can process them. Specific details will not be elaborated here.

[0025] Example 1 Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating a batch task buffer scheduling method based on a three-level hierarchical architecture disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of a three-level hierarchical architecture disclosed in an embodiment of the present invention. Figure 1 The batch task buffer scheduling method based on a three-level hierarchical architecture described herein is applied in distributed computing and task scheduling devices, such as local servers or cloud servers used for distributed computing and task scheduling, and the embodiments of the present invention are not limited thereto.

[0026] A batch task buffer scheduling method based on a three-level hierarchical architecture is proposed, wherein the batch task buffer scheduling method is implemented based on a global scheduling layer (Master), a node management layer (Agent), and a computation execution layer (Engine); for example Figure 2 As shown, the global scheduling layer Master includes one global scheduling center for global scheduling and task buffering; the node management layer Agent includes several node management units for implementing closed-loop control and task status aggregation at the node management level; the computing execution layer Engine includes several sets of computing execution units for computing and executing data processing tasks. It should be noted that a one-to-many bidirectional communication connection is established between the global scheduling center of the global scheduling layer Master and several node management units of the node management layer; a one-to-one bidirectional communication connection is established between several node management units of the node management layer and several groups of computing execution units of the computing execution layer when computing and executing data processing tasks.

[0027] It should be noted that the aforementioned global scheduling center is generally an independent physical server or a highly available primary / backup server, with hardware configuration emphasizing high stability, high I / O, and large memory; the aforementioned node management unit can be a rack server or a worker node machine; the aforementioned computing execution unit group can be a CPU core group, process resource group, container group, or virtual machine on a physical machine, which are hardware / virtualization resources for executing task computing, emphasizing high computing power and multi-core parallelism.

[0028] like Figure 1 As shown, the batch task buffer scheduling method includes: S1, Obtain a data processing task set; the data processing task set includes N data processing tasks, where N is a positive integer; It should be noted that the aforementioned data processing tasks are generally computationally intensive, short-term tasks designed for high-throughput computing scenarios. They are characterized by batch processing, statelessness, distributed nature, standardization and verification capabilities, batch execution, and state tracking. They are suitable for scenarios such as large-sample simulation, scientific computing, batch data processing, machine learning training, and edge computing. These data processing tasks exist as independent, fine-grained tasks with no strong coupling dependencies. Common data processing task types include simulation, computation, ETL, and machine learning. Each data processing task includes a globally unique identifier, task configuration information, a set of execution parameters, runtime environment requirements, data input / output information, execution timeout and resource constraints, and lifecycle status identifiers. These information is used to support task validity verification, batch buffering, node allocation, execution control, state monitoring, and closed-loop scheduling.

[0029] It should be noted that the set of execution parameters for data processing tasks includes execution template legality indicators, data path legality indicators, CPU resource constraint indicators, memory resource constraint indicators, and execution timeout constraint indicators. These indicators comprehensively characterize the scheduling feasibility and execution compliance of data processing tasks from four aspects: completeness, effectiveness, rationality, and security. They are the key basis for judging whether a data processing task is legal and whether it can enter the batch buffering and allocation process.

[0030] S2, the global scheduling layer processes the data processing task set and the node management layer to obtain a valid data processing task set, M target node management units, several batch task packages, and several batch task packages and their mapping relationships with the M target node management units; the valid data processing task set includes R data processing tasks, where R is a positive integer and R≤N; It should be noted that the above processing achieves the removal of abnormal tasks, the screening of healthy nodes, and the packaging of batch tasks, which significantly reduces the network overhead and scheduling pressure caused by fine-grained tasks. These are key processing steps to improve task throughput and resource utilization.

[0031] S3, the global scheduling layer distributes the batch task packages to the M target node management units based on the mapping relationship between the batch task packages and the M target node management units; S4, the M target node management units use the computing execution layer to process the plurality of batch task packages to obtain a plurality of batch task processing status information packages; It should be noted that the aforementioned batch task processing status information package is a set of structured status data encapsulated by the batch dimension when each target node management unit calls the adapted computing execution unit group to compute and execute data processing tasks. Specifically, it includes the following: Batch basic identification information, including batch task package number and corresponding target node management unit ID, is used by the node management layer to classify and summarize by node and batch; Task details and status information include the globally unique identifier of the corresponding data processing task, execution status (pending execution / in execution / completed / failed / abnormal / abnormal code), execution start and end time, and real-time execution progress. These are used by the node management layer to deduplicate, merge, and update the latest status according to the task dimension. Batch statistics include the total number of data processing tasks in this batch, the number of completed tasks, the number of tasks in progress, the number of failed tasks, and the number of exceptions, which are used by the node management layer to quickly analyze the overall execution status of the nodes. Exception and result information, including exception codes, error descriptions, and execution result output paths for data processing tasks, is used by the node management layer to uniformly collect exception information and form a global exception list; Timing information, including batch distribution time, start execution time, and status reporting time, is used by the node management layer to form a complete timing state chain.

[0032] S5, the node management layer processes the plurality of batch task processing status information packets to obtain task processing status aggregate information; It should be noted that the above processing involves the node management layer uniformly aggregating, organizing, merging, and statistically analyzing several batch task processing status information packages to obtain globally unified task processing status aggregated information. The specific aggregation process is as follows: First, each target node management unit deduplicates and verifies the received batch task processing status information packages, primarily filtering out duplicate reports, abnormal formats, and invalid status data. Then, the node management layer merges the filtered batch task processing status information packages by task dimension, mainly using the globally unique identifier of the data processing task as an index to merge multiple status updates of the same task into the latest status. Next, the node management layer summarizes by batch dimension, mainly summarizing and statistically analyzing the execution results, progress, and exception information of all tasks within the same batch task package. Finally, the node management layer performs global status integration, integrating the status data of multiple target nodes and multiple batch tasks into a complete and structured node-level global task processing status data, i.e., task processing status aggregated information. This is generally in tabular form, but can also be other forms of structured data. An example of tabular task processing status aggregated information is shown below: S6, the node management layer reports the task processing status aggregation information to the global scheduling layer; S7, the global scheduling layer determines, based on the task processing status aggregation information, whether all R data processing tasks in the legal data processing task set have been executed, and obtains a first judgment result; When the first judgment result is yes, the batch task buffer scheduling process ends; If the first judgment result is negative, execute S8; S8, the global scheduling layer updates the legitimate data processing task set based on the task processing status aggregation information to obtain the updated data processing task set, and determines the updated data processing task set as the data processing task set, and executes S2.

[0033] As can be seen, the batch task buffer scheduling method based on a three-level hierarchical architecture described in this embodiment of the invention utilizes a three-level hierarchical architecture including a global scheduling layer (Master), a node management layer (Agent), and a computation execution layer (Engine) to batch package, target, distribute, collect status, aggregate in layers, and schedule in a closed loop for data processing task sets. This achieves efficient task distribution, parallel execution, and unified status management, ensuring a stable, complete, and error-free scheduling process. It is adaptable to large-scale computing scenarios with high throughput and low latency requirements.

[0034] In another optional embodiment, the step of using the global scheduling layer to process the data processing task set and the node management layer to obtain a valid data processing task set, M target node management units, several batch task packages, and several batch task packages and their mapping relationships with the M target node management units includes: S21, the global scheduling layer uses the task verification model and parameter verification model to perform task verification and parameter verification on the data processing task set to obtain a valid data processing task set; the valid data processing task set includes R data processing tasks, where R is a positive integer and R≤N. The expression for the task verification model is: In the formula, The task verification value is the i-th data processing task in the data processing task set, and its value range is 0 or 1. This refers to the i-th data processing task in the data processing task set, where i = 1, 2, ..., N; For task identification integrity indicators, It is the globally unique identifier of the i-th data processing task in the data processing task set; As an indicator of the legality of the task status, This represents the initial state of the i-th data processing task in the data processing task set. As an indicator of the rationality of task resources, The amount of computing resources required for the i-th data processing task in the data processing task set; As an indicator of the effectiveness of task types, The business type of the i-th data processing task in the data processing task set; The expression for the parameter verification model is: In the formula, The parameter verification value is for the i-th data processing task in the data processing task set, and its value range is 0 or 1; The set of execution parameters for the i-th data processing task in the data processing task set; To implement the template validity index, The execution configuration template for the i-th data processing task in the data processing task set; This is an indicator of data path validity. The input data path for the i-th data processing task in the data processing task set. The output path of the i-th data processing task in the data processing task set; This is a CPU resource constraint indicator. The number of CPU cores required for the i-th data processing task in the data processing task set. The maximum CPU limit allowed by the system; As a memory resource constraint indicator, The memory size required for the i-th data processing task in the data processing task set. The maximum memory limit allowed by the system; To enforce timeout constraints, The estimated timeout for the execution of the i-th data processing task in the data processing task set. This is the minimum legal timeout time for the system; The expression for the set of legitimate data processing tasks is: In the formula, LegalTask ​​is the set of legal data processing tasks; R is the number of data processing tasks in the set of legal data processing tasks; It should be noted that by using the dual screening of task verification and parameter validation, invalid, illegal, and unqualified tasks are removed from the original task set, forming a stable, schedulable, and securely executable set of legitimate data processing tasks. This effectively reduces the error rate in subsequent scheduling, buffering, and execution stages, reduces invalid network transmission and resource consumption, and improves overall scheduling efficiency and system robustness.

[0035] S22, process the legal data processing task set and the node management layer to obtain M target node management units; S23, Based on the M target node management units, obtain the M buffers and the mapping relationship between the M buffers and the M target node management units; It should be noted that the global scheduling layer creates or reuses corresponding independent task buffers for the M target node management units according to a one-to-one exclusive mapping rule, thus obtaining M buffers that correspond one-to-one with the M target node management units. S24, using the M target node management units, the R data processing tasks of the legal data processing task set are allocated and processed to obtain M task queues and the mapping relationship between the M task queues and the M target node management units; each task queue includes several data processing tasks, and the sum of the queue lengths of the M task queues is R; S25, Based on the mapping relationship between the M task queues and the M target node management units and the mapping relationship between the M buffers and the M target node management units, the mapping relationship between the M task queues and the M buffers is obtained; S26, using the M buffers and the mapping relationship between the M task queues and the M buffers, perform dual-trigger batch buffering processing on the M task queues synchronously to obtain several batch task packages and several batch task packages and the mapping relationship between the M buffers. S27. Based on the mapping relationship between the batch task packages and M buffers and the mapping relationship between the M buffers and M target node management units, the mapping relationship between the batch task packages and M target node management units is obtained.

[0036] As can be seen, the batch task buffer scheduling method based on a three-level hierarchical architecture described in this embodiment of the invention achieves pre-screening of data processing tasks through task verification and parameter verification. Combined with node screening, configuration of dedicated buffers, polling task allocation, dual-trigger batch buffering, and multi-level mapping relationship construction, a scheduling link of task-node-buffer registration mapping is formed. This not only eliminates invalid tasks from the source and avoids execution risks, but also significantly reduces the network interaction overhead of fine-grained tasks through batch buffering and targeted mapping, improving resource utilization and scheduling throughput. At the same time, it provides a standardized and traceable scheduling foundation for subsequent distributed execution, state aggregation, and closed-loop scheduling, ensuring the efficiency, stability, and controllability of batch task scheduling under the three-level hierarchical architecture.

[0037] In another optional embodiment, the processing of the legitimate data processing task set and the node management layer to obtain M target node management units includes: S221, Process the set of legal data processing tasks to obtain the required number M of node management units; It should be noted that the required number M of node management units needs to be determined by taking into account factors such as the number of data processing tasks in the above-mentioned legitimate data processing task set, the complexity and computing power requirements of various data processing tasks, and the maximum number of tasks that a single node management unit can handle.

[0038] S222, a heartbeat detection request is sent to the node management layer, and a heartbeat response message is received from the node management layer; It should be noted that the aforementioned heartbeat detection request is a periodic status detection mechanism initiated by the global scheduling layer to all node management units in the node management layer. By actively sending standardized detection requests, it determines the network connectivity, service operation status, and computing power availability of the node management units in real time. It is a core means of perceiving node health in a distributed architecture and provides objective status basis for subsequent node screening.

[0039] It should be noted that the heartbeat response message is a structured status data message that the node management unit sends back in a preset format after receiving the heartbeat detection request from the global scheduling layer. It contains core status information such as the node's unique identifier, current CPU / memory load rate, service status code, network latency, and the length of the current queue of tasks waiting to be processed. It is the direct data source for the global scheduling layer to determine whether the node is healthy and whether it has the ability to execute tasks.

[0040] It should be noted that, based on the aforementioned periodic state detection mechanism, the zombie node management unit can be cleaned up. An exemplary implementation of this zombie node management unit cleanup is shown below, with its core code as follows: cleanupLoop(): Execute every minute in a loop: Mutex (locking) now ← Current timestamp() activeIDs ← Create New List() ttl ← cfg.DeadAgentTTLSec / / Clean up zombie nodes For each agentID in AgentIDs: node ← Agents[agentID] If now - node.LastActive <ttl: activeIDs. Append (agentID) otherwise: Agents.Delete(agentID) Log the following: ("Clean up zombie nodes: " + agentID") AgentIDs ← activeIDs Unlock (mutex) S223, based on the preset node management unit health status determination rules and the heartbeat response message, the health status of several node management units in the node management layer is screened to obtain a candidate set of healthy node management units; the candidate set of healthy node management units includes L healthy node management units; the health status expression of the node management unit is: In the formula, This represents the health status of the j-th node management unit in the node management layer, with a value of 0 or 1. =1 indicates that the j-th node management unit of the node management layer is a healthy node. =0 indicates that the j-th node management unit of the node management layer is an unhealthy node; The network delay in the heartbeat response message of the j-th node management unit of the node management layer. The preset maximum allowable network latency; The service status code in the heartbeat response message of the j-th node management unit of the node management layer; The CPU real-time load rate in the heartbeat response message of the j-th node management unit of the node management layer. This is the preset maximum CPU load threshold; The memory usage rate is the real-time rate in the heartbeat response message of the j-th node management unit of the node management layer. This is the preset maximum memory usage threshold; The length of the current pending task queue of the j-th node management unit in the node management layer; It should be noted that the above-mentioned node management unit health status determination rules are a comprehensive judgment criterion integrating four dimensions: network connectivity, service operation status, resource load level, and task processing capability. By comparing the quantitative indicators in the heartbeat response message with preset thresholds and performing legality verification on status indicators, the objective and accurate determination of the health status of the node management unit is achieved. Finally, healthy node management units with task execution and management capabilities are selected to form a candidate set of healthy node management units.

[0041] It should be noted that the above judgment rule adopts the judgment logic of "basic status verification plus quantitative indicator threshold verification". Only when the node management unit simultaneously meets the conditions that all basic statuses are valid and all quantitative indicators are within the preset normal range will it be judged as a healthy node; if any dimension is not met, it will be judged as an unhealthy node. The specific judgment dimensions and rules are as follows: Network connectivity determination: Verify the network latency reported in the heartbeat response message. ,when If the network connectivity of the node management unit is normal (e.g., 500ms), the network connectivity is determined to be normal; otherwise, the network is determined to be abnormal. In scenarios with high accuracy requirements, the integrity of the received message can also be checked. A complete message without packet loss or garbled characters is used to determine that the network connectivity is valid; otherwise, the node management unit is determined to be an unhealthy node.

[0042] Service status determination: Extract the service status code from the heartbeat response message. ,when When the system's preset "normal operation code" (such as 0000) is given, the service operation status of the node management unit is determined to be normal; if it is an abnormal code (such as 1001-service suspended, 2002-process abnormal), it is directly determined to be an unhealthy node.

[0043] Resource load level determination: Extract CPU load rate and memory load rate from heartbeat response messages. When the conditions are met... and If the resource load of the node management unit is within the normal range and has sufficient resources to handle new tasks, then the node management unit is deemed to be overloaded and classified as an unhealthy node. If either the CPU load value or the memory utilization rate exceeds the threshold, the node management unit is deemed to be overloaded and classified as an unhealthy node.

[0044] Task processing capacity assessment: Extract the length of the current pending task queue from the heartbeat response message. ,when If the node is currently idle and has normal task reception and processing capabilities, it is determined that the node management unit is currently processing other data processing tasks and is classified as an unhealthy node for this batch of data processing tasks.

[0045] S224, determine whether the number L of the health node management units is greater than or equal to the required number M of the node management units, and obtain a second judgment result; If the second judgment result is yes, execute S225; If the second judgment result is negative, execute S226; S225, Based on the candidate set of health node management units, M target node management units are obtained; It should be noted that when L≥M, the global scheduling layer evaluates the above L healthy node management units based on a comprehensive principle of prioritizing load balancing, adapting performance to tasks, reserving resources for adaptation, and weighting historical performance. This evaluation can be done using methods such as the analytic hierarchy process, entropy weighting, fuzzy comprehensive evaluation, and machine learning evaluation based on historical data. Then, M target node management units are selected in descending order of evaluation scores to ensure that the selected nodes can efficiently handle subsequent task allocation and batch buffering, while also achieving balanced utilization of computing resources among nodes, thus avoiding problems such as single-node overload and task execution delays.

[0046] S226, Create ML new node management units, and integrate the ML node management units and the L healthy node management units to obtain the M target node management units.

[0047] It should be noted that the node management layer has the core function of dynamically creating new node management units according to data processing needs. When it is determined that the number of healthy node management units L is less than the number of nodes required M, the node management layer will respond to the node replenishment instruction of the global scheduling layer, start the node management unit creation process, and accurately create ML node management units (i.e. the number of nodes in shortfall). The creation process strictly follows the system's preset node initialization specifications to ensure that the newly created nodes have the same hardware computing power standards, software operating environment and scheduling adaptation capabilities as the existing healthy node management units.

[0048] As can be seen, implementing the batch task buffer scheduling method based on a three-level hierarchical architecture described in this invention scientifically determines the required number of node management units, achieving precise matching between node resources and the scale and complexity of legitimate data processing tasks, avoiding resource waste and computing power gaps; through heartbeat detection and multi-dimensional node health status judgment, healthy nodes are accurately screened and abnormal nodes are eliminated, reducing task execution risks; through node quantity judgment and new replenishment mechanisms, it ensures that the target number of nodes meets the requirements and fills the gap in healthy nodes. The overall process is standardized and traceable, providing stable support for subsequent task allocation and buffer management, improving the reliability, resource utilization, and overall controllability of batch task scheduling.

[0049] In another optional embodiment, processing the set of legitimate data processing tasks to obtain the required number M of node management units includes: The valid data processing task set is processed using the node management unit demand quantity calculation model to obtain the node management unit demand quantity M; the expression of the node management unit demand quantity calculation model is: In the formula, M is the required number of node management units; R is the number of data processing tasks in the legal data processing task set; and K is the number of data processing task types in the legal data processing task set. Let be the complexity weight of the k-th type of data processing task in the set of legal data processing tasks, with a value range of (0,1). ; γ is the average computing power coefficient of the k-th type of data processing task in the set of legal data processing tasks; Q is the preset maximum number of baseline data processing tasks that a single node management unit can carry; β is the load balancing coefficient of the node management unit, with a value range of [0.7, 0.9]; γ is the dynamic redundancy adjustment factor, with a value range of [0.1, 0.3]. The preset maximum number of legal tasks that the system can support; It should be noted that, through the above-mentioned node management unit requirement calculation model, the minimum node size required to support the execution of legitimate tasks can be obtained while taking into account both computing power and scheduling efficiency. Within the framework of this invention, the number of data processing task types K=4 in the legitimate data processing task set, corresponding to four types of data processing tasks: simulation, computation, ETL, and machine learning. The number of types can also be increased based on the actual data task situation, and the model remains applicable. The complexity weight of the k-th type of data processing task in the legitimate data processing task set... This complexity weight is used to differentiate the computing power requirements of different types of tasks, avoiding node adaptation bias caused by single-dimensional calculations. It is determined through normalization based on the average computing power consumption, I / O intensity, execution time percentage, and task type priority of a single historical execution of that type of data processing task. The average computing power coefficient of the k-th type of data processing task in the set of legitimate data processing tasks... This is used to characterize the average computing power consumption of a single k-th type of data processing task, adapting to the differences in resource requirements of different data processing tasks. The value was determined by calculating the average of computing power-related indicators such as CPU utilization, memory consumption, and computation time during the historical execution of this type of data processing task, combined with the normalized ratio of the benchmark computing power task (set to 1.0).

[0050] It should be noted that the preferred value of the load balancing coefficient β of the node management unit is 0.8; and the preferred value of the dynamic redundancy adjustment factor γ is 0.2.

[0051] As can be seen, the batch task buffer scheduling method based on a three-level hierarchical architecture described in this embodiment of the invention utilizes a node management unit demand quantity calculation model to determine the required number M of node management units, providing a scientific basis for the number of nodes for batch task buffer scheduling. This effectively solves the problems of subjective quantity estimation and mismatch with task requirements in traditional node allocation. This calculation model comprehensively considers the total number of tasks in a valid task set, the complexity of various tasks, and computing power requirements, ensuring that the number of nodes is accurately matched with the task scale and computing power requirements. This avoids task backlog and execution delays due to insufficient nodes, or waste of computing resources caused by node redundancy. Simultaneously, the model incorporates a load balancing coefficient to ensure load balancing across nodes, preventing single-node overload and guaranteeing task execution stability. A dynamic redundancy adjustment factor is introduced to flexibly adjust the redundancy amount based on the system's maximum supported task volume, improving the system's ability to cope with task fluctuations.

[0052] In another optional embodiment, the step of using the M target node management units to allocate and process the R data processing tasks of the legitimate data processing task set to obtain M task queues and the mapping relationship between the M task queues and the M target node management units includes: S241, Based on the M target node management units, a target node management unit sequence list is obtained; the length of the target node management unit sequence list is M. It should be noted that the above target node management unit sequence list can be sorted in descending order according to the node comprehensive screening score, or in ascending order according to the node identifier.

[0053] S242, Based on the target node management unit sequence table, establish M corresponding empty task queues to obtain the mapping relationship between the M empty task queues and the M target node management units; It should be noted that setting up the corresponding M empty task queues is a prerequisite for task allocation, and the core is to establish a one-to-one correspondence between "nodes" and "queues".

[0054] S243, Based on the set of legal data processing tasks, a queue of data processing tasks to be assigned is obtained; the length of the queue of data processing tasks to be assigned is R; It should be noted that the core of this step is to arrange the above-mentioned set of legal data processing tasks in an orderly manner according to the preset task sorting rules (such as task priority, task complexity, and task submission time) to obtain the above-mentioned queue of data processing tasks to be assigned.

[0055] S244, based on the mapping relationship between the M empty task queues and the M target node management units, process the data processing task queue to be assigned and the M empty task queues to obtain the M task queues and the mapping relationship between the M task queues and the M target node management units.

[0056] It should be noted that the above processing of the data processing task queue to be allocated and the M empty task queues is performed according to a preset round-robin load balancing strategy to enqueue tasks. Specifically, the order of the target node management unit sequence list is used as the polling order. Starting from the first task in the data processing task queue to be allocated, each task is sequentially allocated to the M empty task queues corresponding to the M target node management units. That is, the first task is allocated to the task queue corresponding to the first target node in the sequence list, the second task is allocated to the task queue corresponding to the second target node, and so on. When the task is allocated to the task queue corresponding to the Mth target node, the next task is re-allocated from the task queue corresponding to the first target node. This process is repeated until all R tasks in the data processing task queue to be allocated are allocated, resulting in M ​​task queues containing specific tasks. At the same time, the one-to-one mapping relationship between the M task queues and the M target node management units is maintained and confirmed to ensure that the task allocation is even and to achieve load balancing among the target node management units, providing support for subsequent task buffering and batch distribution.

[0057] It should be noted that the above-described exemplary implementation of task enqueuing according to a preset round-robin load balancing strategy is shown below, and its core code is as follows: Input: None Output: Selected Agent node Method SelectAgent(): Mutex (locking) count ← AgentIDs.size() If count = 0: Returns null startIndex ← nextAgent % count For i from 0 to count-1: idx ← (startIndex + i) % count agentID ← AgentIDs[idx] node ← Agents[agentID] threshold ← cfg.HeartbeatThresholdSec / / Check if the Agent is active If current timestamp() - node.LastActive ≤ threshold: nextAgent ← (idx + 1) % count / / Update polling pointer Unlock (mutex) Return node Unlock (mutex) Returns null / / No Agent available As can be seen, the batch task buffer scheduling method based on a three-level hierarchical architecture described in the embodiments of the present invention realizes the polling matching and balanced queuing of R legal data processing tasks with M target node management units, effectively avoids the problem of single-node task backlog, ensures load balance of each node, and provides a task queue with a clear structure and clear mapping for subsequent dual-trigger batch buffer processing.

[0058] In another optional embodiment, the step of using the M buffers and the mapping relationship between the M task queues and the M buffers to synchronously perform dual-trigger batch buffering processing on the M task queues to obtain several batch task packages and several batch task packages and their mapping relationship with the M buffers includes: S261, Start the preset M timers to begin recording the task buffer duration of the corresponding buffers; S262, based on the mapping relationship between the M task queues and the M buffers, the data processing tasks of the M task queues are continuously input into the corresponding M buffers one by one according to a preset time step. S263, determine whether the number of data processing tasks in the M buffers has reached a preset number threshold, and obtain a third judgment result; If the third determination result is yes, execute S265; If the result of the third judgment is negative, execute S264; It should be noted that the above quantity threshold is generally set to 50.

[0059] S264, determine whether the task buffer duration has reached the preset duration threshold, and obtain the fourth determination result; If the fourth determination result is yes, execute S265; If the result of the fourth judgment is negative, execute S262; It should be noted that the above duration threshold is generally set to 200ms.

[0060] S265, the data processing tasks in the buffer where the third judgment result is yes and / or the fourth judgment result is yes are packaged and output, and the buffer where the third judgment result is yes and / or the fourth judgment result is yes and the corresponding timer are reset; S266, repeat S262 to S265 until the length of all M task queues is 0, to obtain several batch task packages and several batch task packages and M buffer mapping relationships.

[0061] As can be seen, the batch task buffering scheduling method based on a three-level hierarchical architecture described in this embodiment of the invention achieves high efficiency, balance, and controllability of task buffering by implementing a dual-trigger batch buffering strategy to obtain several batch task packages and their mapping relationships with M buffers. Relying on a timer synchronization timing and continuous task input mechanism, the synchronous buffering of the M buffers and their corresponding task queues is ensured, avoiding task backlog or buffer delays. The dual-trigger mechanism of "quantity threshold + duration threshold" ensures batch output after the buffered tasks reach a certain scale, improving task distribution and execution efficiency, while avoiding frequent distribution due to insufficient task volume, reducing network interaction overhead. Through buffered package output and timely reset of buffers and timers, the cyclical and efficient execution of task buffering is achieved, ensuring that all tasks in the M task queues can complete buffering processing in an orderly manner. Finally, batch task packages and their corresponding mapping relationships are obtained, providing standardized and batch support for subsequent task distribution to target nodes, further improving the overall scheduling efficiency, stability, and resource utilization.

[0062] In another optional embodiment, the M target node management units utilize the computation execution layer to process the plurality of batch task packages, obtaining a plurality of batch task processing status information packages, including: S41, the M target node management units preprocess the batch task packages they receive to obtain M local task queues. It should be noted that the above preprocessing of batch task packages includes task request parsing, overload checking, configuration merging, and secondary sorting of task priorities. This integrates the scattered batch task packages received by each node management unit into an ordered, directly executable local task queue, forming a total of M local task queues that correspond one-to-one with the target node management unit.

[0063] It should be noted that task request parsing is an operation that performs structured parsing of each data processing task request within a batch task package. The core is to extract the core execution information from the task request, including the unique task identifier, execution instructions, input and output paths, computing power requirement parameters, execution dependencies, etc., and convert the valid task request into a standardized data structure that can be recognized by the computing execution layer. This removes format obstacles for subsequent task execution and ensures that the task instructions can be accurately recognized and responded to by the computing execution unit.

[0064] It should be noted that the overload check is an operation that assesses the overall computing power load of the batch task packages to be processed, based on the target node management unit's own computing power capacity and current operating status. The core is to calculate the total computing power requirement and estimated execution time of the batch task packages, and compare them with the node's remaining computing power and idle execution resources. If the total load of the task packages exceeds the node's current carrying capacity threshold, a task temporary storage or sharding processing mechanism is triggered. If it is within the carrying capacity range, processing is allowed to continue. This check avoids computing power overload and service lag caused by the node taking on too many tasks, ensuring the stable operation of the target node management unit and the subsequent computing execution layer.

[0065] It should be noted that configuration merging is an operation that unifies and standardizes the configuration information of each data processing task within a batch task package. The core is to sort out the common configurations (such as system environment configuration, resource access configuration, log output configuration, etc.) and personalized configurations of each task, load the common configurations globally, mark the personalized configurations separately and bind them to the corresponding tasks, and complete the default configuration parameters required for task execution. This eliminates configuration conflicts between different tasks, forming a unified and standardized task execution configuration system, ensuring that the configuration information of each task is complete and conflict-free during execution, and improving the smoothness of task execution.

[0066] It should be noted that the secondary priority sorting of tasks is an operation that re-priorities the valid tasks within the batch task package based on the initial priority of the global scheduling layer and the local execution needs and resource status of the target node management unit. The core is to comprehensively consider local adaptation factors such as the urgency of tasks, the amount of computing power required, the estimated execution time, and resource dependencies, and to re-divide the task execution priorities and arrange them in an orderly manner. Tasks with high urgency, low computing power requirements, and no resource dependencies are given priority to be placed at the top, so as to achieve rational scheduling of local tasks, improve the utilization efficiency of node computing power resources, and avoid low execution efficiency or delays of critical tasks due to unreasonable task sorting.

[0067] S42, the M target node management units, based on the M local task queues, filter the computing execution layer to obtain M computing execution unit groups and the mapping relationship between the M target node management units and the M computing execution unit groups; each computing execution unit group includes several computing execution units; It should be noted that each target node management unit, based on the characteristics of its local task queue such as task size, computing power requirements, and execution type, selectively filters all computing execution units in the computing execution layer. Following the principles of "computing power matching, type adaptation, and load balancing," each target node management unit is matched with a dedicated computing execution unit group consisting of several computing execution units, resulting in M ​​computing execution unit groups. At the same time, a one-to-one mapping relationship between the M target node management units and the M computing execution unit groups is established and stored, clarifying the scheduling association between nodes and execution unit groups. This ensures that the computing power resources of the computing execution layer are accurately matched with the task execution requirements of the nodes, thereby improving the utilization rate of computing power resources.

[0068] S43, the M target node management units, based on the mapping relationship between the M target node management units and the M computing execution unit groups, respectively call the corresponding computing execution unit groups to synchronously calculate and execute the M local task queues to obtain several task processing status information; It should be noted that, based on the above mapping relationship, the M target node management units issue task execution instructions to their respective computing execution unit groups, triggering all computing execution unit groups to synchronously and in parallel execute the data processing tasks in their corresponding local task queues; after completing the execution of a single / batch task, each computing execution unit provides real-time feedback of independent task processing status information (including task ID, execution result, success / failure flag, execution time, etc.), ultimately forming several fine-grained status information corresponding to a single / batch task, realizing distributed parallel processing of task execution and significantly improving the overall task execution efficiency.

[0069] S44, based on the intelligent status batch reporting mechanism, the processing status information of the several tasks is processed to obtain several batch task processing status information packages.

[0070] It should be noted that the aforementioned intelligent batch status reporting mechanism involves each computing execution unit group packaging the aforementioned task processing status information according to the real-time generation order of the information, based on preset packaging and reporting trigger conditions, and then reporting it to the corresponding target node management unit. This intelligent batch status reporting mechanism achieves batch aggregation and standardized reporting of fine-grained status information, significantly reducing the number of network interactions for status feedback, lowering communication overhead, and facilitating unified management and traceability of the execution status of batch tasks by the global scheduling layer.

[0071] As can be seen, the batch task buffer scheduling method based on a three-level hierarchical architecture described in this embodiment of the invention achieves precise matching of computing power, efficient parallel execution, and intensive feedback of status in the task execution stage through the preprocessing of batch task packages by the node management layer, precise screening and mapping of the computing execution layer, distributed parallel execution, and intelligent batch reporting of status. It ensures the standardization of task execution and optimal matching of computing resources through preprocessing and computing power screening, avoiding computing power waste and execution adaptation deviation. It also significantly improves the overall processing efficiency of batch tasks by relying on the synchronous parallel execution of computing execution unit groups, adapting to the needs of large-scale data processing. At the same time, through the intelligent batch reporting mechanism, fine-grained status information is aggregated into standardized batch packages, reducing the network interaction overhead of status feedback, realizing unified management and traceability of execution status, and constructing an efficient collaborative execution closed loop between the node management layer and the computing execution layer, further improving the execution efficiency, resource utilization, and status management capabilities of batch task buffer scheduling.

[0072] In another optional embodiment, the intelligent status batch reporting mechanism processes the plurality of task processing status information to obtain a plurality of batch task processing status information packages, including: S441, Preset task processing status information queue; the initial state of the task processing status information queue is an empty queue; S442, add the several task processing status information to the task processing status information queue one by one in the order they are generated in real time. S443, determine whether the length of the task processing status information queue is greater than or equal to a preset length threshold, and obtain the fifth judgment result; If the fifth judgment result is yes, execute S446; If the result of the fifth judgment is negative, execute S444; It should be noted that the aforementioned length threshold is a pre-defined threshold for the number of task processing status information queues to be stored, based on the task processing speed of the computing execution layer, network communication bandwidth, and the status processing capability of the node management layer. It is a positive integer, and its value comprehensively considers the balance between the efficiency and real-time performance of batch reporting. It avoids the situation where the threshold is too small, which would lead to frequent single reporting of status information and increase network communication overhead, and it also avoids the situation where the threshold is too large, which would lead to long-term caching of status information and reduce the real-time control of task execution status by the node management layer. It is the core conventional quantitative condition for triggering the batch packaging and reporting of status information. When the number of status information stored in the queue reaches or exceeds this threshold, the packaging and reporting operation will be directly triggered.

[0073] S444, Determine whether there is a batch mode disabled flag in the task processing status information queue, and obtain the sixth determination result; If the result of the sixth judgment is yes, execute S446; If the result of the sixth judgment is negative, execute S445; It should be noted that the aforementioned batch mode disable flag is a standardized status trigger flag that the system actively writes to the task processing status information queue under global abnormal scenarios. It is a preset character / numeric flag, and its triggering scenarios include computing power overload at the computing execution layer, unstable communication links between nodes and computing execution units, system scheduling mode switching, and emergency status feedback instructions issued by the node management layer. The presence of this flag indicates that the current system is no longer suitable for continuing to cache status information for batch reporting. It is a global abnormal condition that triggers the immediate packaging and reporting of status information. Regardless of whether the queue length has reached the length threshold, detecting this flag will force the packaging and reporting to be triggered, ensuring timely feedback of status information under abnormal scenarios and avoiding data delays or loss.

[0074] S445, determine whether there is a separate reporting identifier in the task processing status information queue, and obtain the seventh judgment result; If the result of the seventh judgment is yes, execute S446; If the seventh judgment result is negative, then it is determined whether all the task processing status information has been enqueued to obtain the eighth judgment result; When the eighth judgment result is yes, the task processing status information queue is packaged to obtain a batch task processing status information package, and then S5 is executed; If the result of the eighth judgment is negative, execute S446; It should be noted that the aforementioned separate reporting identifier is a standardized trigger identifier written into the task processing status information queue along with the processing status information of high-priority / special scenario tasks. It is a preset character / numeric identifier, and the task types it is bound to include core business tasks, execution failure / timeout / abnormal tasks, and user-specified emergency monitoring tasks. The existence of this identifier indicates that the queue contains critical task status information that the node management layer needs to know in real time. It is an emergency condition for triggering the immediate packaging and reporting of status information for a single task. There is no need to wait for the queue length to reach the threshold. Detecting this identifier will trigger the packaging and reporting, ensuring real-time traceability and rapid response to the execution status of critical tasks, and balancing the efficiency of batch reporting with the real-time control of core tasks.

[0075] S446, The task processing status information queue is packaged to obtain a batch task processing status information package. S447, the batch task processing status information packet is sent to the corresponding target node management unit, and the task processing status information queue is cleared. S448, update the task processing status information queue to obtain the updated task processing status information queue, and confirm the updated task processing status information queue as the task processing status information queue, then execute S442.

[0076] As can be seen, the batch task buffer scheduling method based on a three-level hierarchical architecture described in this embodiment of the invention achieves intelligent batch reporting of task processing status information through a full-process design that includes a preset cache queue, orderly queuing of stored status information, multi-condition hierarchical triggering, and standardized packaged reporting. Relying on the conventional triggering mechanism of the task processing status information queue length threshold, the scattered status information is integrated and reported in batches, significantly reducing the number of network communication interactions, lowering system communication overhead and node processing pressure, and improving status reporting efficiency. Through the triggering mechanism of batch mode disabling flags and individual reporting flags, mandatory reporting of existing task processing status information in abnormal scenarios and emergency reporting of high-priority task processing status information are achieved. The aforementioned multi-condition hierarchical triggering strategy balances the efficiency of batch reporting with the real-time nature of special scenarios, avoiding delays or loss of status information. Simultaneously, through the design of orderly queue storage, precise packaged sending, and clearing and resetting after reporting, the timing, integrity, and closed-loop circulation of status information are ensured, providing accurate and timely status data support for the node management layer to monitor task execution status in real time, quickly handle abnormal tasks, and optimize computing power scheduling strategies.

[0077] Example 2 Please see Figure 3 , Figure 3 This is a schematic diagram of a batch task buffer scheduling device based on a three-level hierarchical architecture disclosed in an embodiment of the present invention. Figure 3 The batch task buffer scheduling device based on a three-level hierarchical architecture described herein can be applied to distributed computing and task scheduling devices, local servers, or cloud servers, etc., and the embodiments of this invention are not limited thereto. Figure 3 As shown, this batch task buffer scheduling device based on a three-level hierarchical architecture may include: Processor 301; A memory 302 containing executable program code is coupled to the processor 301; The processor 301 calls the executable program code stored in the memory 302 to execute some or all of the steps of the batch task buffer scheduling method based on a three-level hierarchical architecture according to Embodiment 1.

[0078] Example 3 This invention discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements some or all of the steps of a batch task buffer scheduling method based on a three-level hierarchical architecture according to Embodiment 1.

[0079] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable rewritable read-only memory (EEPROM), read-only optical disc (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0080] Example 4 This invention discloses an information data processing terminal, which is used to implement some or all of the steps of a batch task buffer scheduling method based on a three-level hierarchical architecture according to Embodiment 1.

[0081] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable rewritable read-only memory (EEPROM), read-only optical disc (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0083] Finally, it should be noted that the batch task buffer scheduling method and apparatus based on a three-level hierarchical architecture disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A batch task buffer scheduling method based on a three-level hierarchical architecture, characterized in that, The method is implemented based on a global scheduling layer (Master), a node management layer (Agent), and a computation execution layer (Engine). The global scheduling layer (Master) includes a global scheduling center for global scheduling and task buffering. The node management layer (Agent) includes several node management units for implementing closed-loop control and task status aggregation at the node management level. The computation execution layer (Engine) includes several groups of computation execution units for computing and executing data processing tasks. The batch task buffer scheduling method includes: S1, Obtain a data processing task set; the data processing task set includes N data processing tasks, where N is a positive integer; S2, the global scheduling layer is used to process the data processing task set and the node management layer to obtain a valid data processing task set, M target node management units, several batch task packages, and several batch task packages and their mapping relationships with the M target node management units; the valid data processing task set includes R data processing tasks, where R is a positive integer and R≤N. S3, the global scheduling layer distributes the batch task packages to the M target node management units based on the mapping relationship between the batch task packages and the M target node management units; S4, the M target node management units use the computing execution layer to process the plurality of batch task packages to obtain a plurality of batch task processing status information packages; S5, the node management layer processes the plurality of batch task processing status information packets to obtain task processing status aggregate information; S6, the node management layer reports the task processing status aggregation information to the global scheduling layer; S7, the global scheduling layer determines, based on the task processing status aggregation information, whether all R data processing tasks in the legal data processing task set have been executed, and obtains a first judgment result; When the first judgment result is yes, the batch task buffer scheduling process ends; If the first judgment result is negative, execute S8; S8, the global scheduling layer updates the legitimate data processing task set based on the task processing status aggregation information to obtain the updated data processing task set, and determines the updated data processing task set as the data processing task set, and executes S2.

2. The batch task buffer scheduling method based on a three-level hierarchical architecture according to claim 1, characterized in that, The process of using the global scheduling layer to process the data processing task set and the node management layer to obtain a valid data processing task set, M target node management units, several batch task packages, and a mapping relationship between the several batch task packages and the M target node management units includes: S21, the global scheduling layer uses the task verification model and parameter verification model to perform task verification and parameter verification on the data processing task set to obtain a valid data processing task set; the valid data processing task set includes R data processing tasks, where R is a positive integer and R≤N. The expression for the task verification model is: In the formula, The task verification value is the i-th data processing task in the data processing task set, and its value range is 0 or 1. This refers to the i-th data processing task in the data processing task set, where i = 1, 2, ..., N; For task identification integrity indicators, It is the globally unique identifier of the i-th data processing task in the data processing task set; As an indicator of the legality of the task status, This represents the initial state of the i-th data processing task in the data processing task set. As an indicator of the rationality of task resources, The amount of computing resources required for the i-th data processing task in the data processing task set; As an indicator of the effectiveness of task types, The business type of the i-th data processing task in the data processing task set; The expression for the parameter verification model is: In the formula, The parameter verification value is for the i-th data processing task in the data processing task set, and its value range is 0 or 1; The set of execution parameters for the i-th data processing task in the data processing task set; To implement the template validity index, The execution configuration template for the i-th data processing task in the data processing task set; This is an indicator of data path validity. The input data path for the i-th data processing task in the data processing task set. The output path of the i-th data processing task in the data processing task set; This is a CPU resource constraint indicator. The number of CPU cores required for the i-th data processing task in the data processing task set. The maximum CPU limit allowed by the system; As a memory resource constraint indicator, The memory size required for the i-th data processing task in the data processing task set. The maximum memory limit allowed by the system; To enforce timeout constraints, The estimated timeout for the execution of the i-th data processing task in the data processing task set. This is the minimum legal timeout time for the system. The expression for the set of legitimate data processing tasks is: In the formula, LegalTask ​​is the set of legal data processing tasks; R is the number of data processing tasks in the set of legal data processing tasks; S22, process the legal data processing task set and the node management layer to obtain M target node management units; S23, Based on the M target node management units, obtain the M buffers and the mapping relationship between the M buffers and the M target node management units; S24, using the M target node management units, the R data processing tasks of the legal data processing task set are allocated and processed to obtain M task queues and the mapping relationship between the M task queues and the M target node management units; each task queue includes several data processing tasks, and the sum of the queue lengths of the M task queues is R; S25, Based on the mapping relationship between the M task queues and the M target node management units and the mapping relationship between the M buffers and the M target node management units, the mapping relationship between the M task queues and the M buffers is obtained; S26, using the M buffers and the mapping relationship between the M task queues and the M buffers, perform dual-trigger batch buffering processing on the M task queues synchronously to obtain several batch task packages and several batch task packages and the mapping relationship between the M buffers. S27. Based on the mapping relationship between the batch task packages and M buffers and the mapping relationship between the M buffers and M target node management units, the mapping relationship between the batch task packages and M target node management units is obtained.

3. The batch task buffer scheduling method based on a three-level hierarchical architecture according to claim 2, characterized in that, The process of processing the legitimate data processing task set and the node management layer yields M target node management units, including: S221, Process the set of legal data processing tasks to obtain the required number M of node management units; S222, a heartbeat detection request is sent to the node management layer, and a heartbeat response message is received from the node management layer; S223, based on the preset node management unit health status determination rules and the heartbeat response message, the health status of several node management units in the node management layer is screened to obtain a candidate set of healthy node management units; the candidate set of healthy node management units includes L healthy node management units; the health status expression of the node management unit is: In the formula, This represents the health status of the j-th node management unit in the node management layer, with a value of 0 or 1. =1 indicates that the j-th node management unit of the node management layer is a healthy node. =0 indicates that the j-th node management unit of the node management layer is an unhealthy node; The network delay in the heartbeat response message of the j-th node management unit of the node management layer. The preset maximum allowable network latency; The service status code in the heartbeat response message of the j-th node management unit of the node management layer; The CPU real-time load rate in the heartbeat response message of the j-th node management unit of the node management layer. This is the preset maximum CPU load threshold; The memory usage rate is the real-time rate in the heartbeat response message of the j-th node management unit of the node management layer. This is the preset maximum memory usage threshold; The length of the current pending task queue of the j-th node management unit in the node management layer; S224, determine whether the number L of the health node management units is greater than or equal to the required number M of the node management units, and obtain a second judgment result; If the second judgment result is yes, execute S225; If the second judgment result is negative, execute S226; S225, Based on the candidate set of health node management units, M target node management units are obtained; S226, Create ML new node management units, and integrate the ML node management units and the L healthy node management units to obtain the M target node management units.

4. The batch task buffer scheduling method based on a three-level hierarchical architecture according to claim 2, characterized in that, The process of allocating and processing the R data processing tasks of the legitimate data processing task set using the M target node management units to obtain M task queues and the mapping relationship between the M task queues and the M target node management units includes: S241, Based on the M target node management units, a target node management unit sequence list is obtained; the length of the target node management unit sequence list is M. S242, Based on the target node management unit sequence table, establish M corresponding empty task queues to obtain the mapping relationship between the M empty task queues and the M target node management units; S243, Based on the set of legal data processing tasks, a queue of data processing tasks to be assigned is obtained; the length of the queue of data processing tasks to be assigned is R; S244, based on the mapping relationship between the M empty task queues and the M target node management units, process the data processing task queue to be assigned and the M empty task queues to obtain the M task queues and the mapping relationship between the M task queues and the M target node management units.

5. The batch task buffer scheduling method based on a three-level hierarchical architecture according to claim 2, characterized in that, The method utilizes the M buffers and the mapping relationship between the M task queues and the M buffers to synchronously perform dual-trigger batch buffering processing on the M task queues, resulting in several batch task packages and several batch task packages mapped to the M buffers, including: S261, Start the preset M timers to begin recording the task buffer duration of the corresponding buffers; S262, based on the mapping relationship between the M task queues and the M buffers, the data processing tasks of the M task queues are continuously input into the corresponding M buffers one by one according to a preset time step. S263, determine whether the number of data processing tasks in the M buffers has reached a preset number threshold, and obtain a third judgment result; If the third determination result is yes, execute S265; If the result of the third judgment is negative, execute S264; S264, determine whether the task buffer duration has reached the preset duration threshold, and obtain the fourth determination result; If the fourth determination result is yes, execute S265; If the result of the fourth judgment is negative, execute S262; S265, the data processing tasks in the buffer where the third judgment result is yes and / or the fourth judgment result is yes are packaged and output, and the buffer where the third judgment result is yes and / or the fourth judgment result is yes and the corresponding timer are reset; S266, repeat S262 to S265 until the length of all M task queues is 0, to obtain several batch task packages and several batch task packages and M buffer mapping relationships.

6. The batch task buffer scheduling method based on a three-level hierarchical architecture according to claim 1, characterized in that, The M target node management units utilize the computation execution layer to process the plurality of batch task packages, obtaining a plurality of batch task processing status information packages, including: S41, the M target node management units preprocess the batch task packages they receive to obtain M local task queues. S42, the M target node management units, based on the M local task queues, filter the computing execution layer to obtain M computing execution unit groups and the mapping relationship between the M target node management units and the M computing execution unit groups; each computing execution unit group includes several computing execution units; S43, the M target node management units, based on the mapping relationship between the M target node management units and the M computing execution unit groups, respectively call the corresponding computing execution unit groups to synchronously calculate and execute the M local task queues to obtain several task processing status information; S44, based on the intelligent status batch reporting mechanism, the processing status information of the several tasks is processed to obtain several batch task processing status information packages.

7. The batch task buffer scheduling method based on a three-level hierarchical architecture according to claim 6, characterized in that, The intelligent status batch reporting mechanism processes the status information of the plurality of tasks to obtain a plurality of batch task processing status information packets, including: S441, Preset task processing status information queue; the initial state of the task processing status information queue is an empty queue; S442, add the several task processing status information to the task processing status information queue one by one in the order they are generated in real time. S443, determine whether the length of the task processing status information queue is greater than or equal to a preset length threshold, and obtain the fifth judgment result; If the fifth judgment result is yes, execute S446; If the result of the fifth judgment is negative, execute S444; S444, Determine whether there is a batch mode disabled flag in the task processing status information queue, and obtain the sixth determination result; If the result of the sixth judgment is yes, execute S446; If the result of the sixth judgment is negative, execute S445; S445, determine whether there is a separate reporting identifier in the task processing status information queue, and obtain the seventh judgment result; If the result of the seventh judgment is yes, execute S446; If the seventh judgment result is negative, then it is determined whether all the task processing status information has been enqueued to obtain the eighth judgment result; When the eighth judgment result is yes, the task processing status information queue is packaged to obtain a batch task processing status information package, and then S5 is executed; If the result of the eighth judgment is negative, execute S446; S446, The task processing status information queue is packaged to obtain a batch task processing status information package. S447, the batch task processing status information packet is sent to the corresponding target node management unit, and the task processing status information queue is cleared. S448, update the task processing status information queue to obtain the updated task processing status information queue, and confirm the updated task processing status information queue as the task processing status information queue, then execute S442.

8. A batch task buffer scheduling device based on a three-level hierarchical architecture, applied to the batch task buffer scheduling method based on a three-level hierarchical architecture as described in any one of claims 1-7, characterized in that, The device includes: processor; A memory coupled to the processor stores executable program code; The processor calls the executable program code stored in the memory to execute the batch task buffer scheduling method based on a three-level hierarchical architecture as described in any one of claims 1-7.

9. A computer-readable storage medium, applied to the batch task buffer scheduling method based on a three-level hierarchical architecture as described in any one of claims 1-7, characterized in that, The computer-readable storage medium stores computer instructions, which, when invoked by a computer, are used to execute the batch task buffer scheduling method based on a three-level hierarchical architecture as described in any one of claims 1-7.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the batch task buffer scheduling method based on a three-level hierarchical architecture as described in any one of claims 1-7.

Images