A system, method, device and medium for multi-cluster task scheduling and monitoring
Through the multi-cluster task scheduling and monitoring system, effective scheduling and load balancing of distributed task processing systems are achieved, the problems of waste of resources and high operation and maintenance costs are solved, and efficient and reliable task execution and real-time monitoring mechanisms are provided.
Patent Information
- Application Number
- CN202411167429.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-23
AI Technical Summary
The existing distributed task processing system cannot effectively perform task scheduling and load balancing when managing multiple clusters, resulting in waste of resources and high operation and maintenance costs, and lack of real-time job operation monitoring and alarm mechanisms, which affects job stability.
Using a multi-cluster task scheduling and monitoring system, by registering cluster modules, activating component modules, starting job modules and allocation modules, task scheduling and allocation based on user requests and inertial data, and issue warnings and reallocating tasks when task processing fails , ensure efficient execution of jobs and stable system.
It improves the execution efficiency of jobs, reduces resource waste and operation and maintenance costs, provides high-reliability and real-time monitoring and alarm mechanisms, ensuring that the system can promptly discover and solve problems when facing hardware failures or network problems.
Smart Images

Figure CN118672758B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a system, method, device and medium for multi-cluster task scheduling and monitoring. Background Art
[0002] Distributed task processing has high computing efficiency because it can process different parts of a task in parallel. It is widely used in big data scenarios, especially in distributed graph databases. In the current distributed task processing process, it is impossible to effectively manage multiple clusters in the "1+N" mode, and jobs cannot be scheduled and load balanced between clusters, resulting in a waste of resources. In addition, a large amount of manpower and material resources are required for operation and maintenance, resulting in high operation and maintenance costs.
[0003] In order to solve the above problems, the patent with patent number CN103207814B proposes a decentralized cross-cluster resource management and task scheduling system and scheduling method. When local resources are insufficient but external cluster resources are sufficient, the task manager distributes tasks and schedules them to other clusters to execute tasks, and retrieves the result files after the tasks are completed. However, this method lacks an effective and real-time job operation monitoring and alarm mechanism. When problems occur in the job, they cannot be discovered and solved in time, thus affecting the stable operation of the job; and it cannot provide high reliability. When facing hardware failures or network problems, the operation of the job may be affected.
[0004] Therefore, a system and method for multi-cluster task scheduling and monitoring are provided to help improve the execution efficiency of jobs. Summary of the invention
[0005] The present invention includes a method for multi-cluster task scheduling and monitoring, which is executed by a server, and includes: based on a user request, sending a registration instruction to at least one cluster to be registered to generate a registered cluster; based on the inertial data volume of the registered cluster, determining at least one job component to be activated, and issuing an activation instruction to the at least one job component to be activated to generate an activated job component set corresponding to the registered cluster, wherein the inertial data volume is the data generation rate within a preset historical period; based on the data volume of a first task in a first preset period, generating at least one start instruction, and sending at least one of the start instructions to the registered cluster; wherein the registered cluster includes a scheduling component and a job component, and the scheduling component is used to receive and schedule the first task to the activated job component for processing based on the start instruction; based on the load state value of the activated job component and the amount of data to be processed, allocating and processing a second task in a second preset period in the activated job component set, wherein the second preset period is a time period after the first preset period; and in response to a task processing failure, issuing a warning based on the failure cause and the failure occurrence time, and reallocating the failed task in the activated job component set, wherein the failed task includes at least one of the failed first task and the failed second task.
[0006] The present invention includes a system for multi-cluster task scheduling and monitoring, the system including: a registration cluster module, configured to send a registration instruction to at least one to-be-registered cluster based on a user request to generate a registered cluster; an activation component module, configured to determine at least one to-be-activated job component based on the inertial data volume of the registered cluster, and send an activation instruction to the at least one to-be-activated job component to generate an activated job component set corresponding to the registered cluster, the inertial data volume being the data generation rate within a preset historical period; a start-up job module, configured to generate at least one start-up instruction based on the data volume of the first task in the first preset period, and send at least one of the start-up instructions to the registered cluster; wherein the The registration cluster includes a scheduling component and a job component, the scheduling component is used to receive and schedule the first task to the activation job component for processing based on the startup instruction; and a first allocation module is configured to allocate and process the second task of a second preset time period in the activation job component set based on the load status value of the activation job component and the amount of data to be processed, and the second preset time period is a time period after the first preset time period; the second allocation module is configured to issue a warning in response to a task processing failure based on the cause of the failure and the time when the failure occurred, and reallocate the failed task in the activation job component set, and the failed task includes at least one of the failed first task and the failed second task.
[0007] The content of the present invention includes a device for multi-cluster task scheduling and monitoring, which includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least part of the computer instructions to implement the multi-cluster task scheduling and monitoring method.
[0008] The present invention includes a computer-readable medium, wherein the medium stores computer instructions. When a computer reads the computer instructions in the medium, the computer executes the multi-cluster task scheduling and monitoring method. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:
[0010] Figure 1 It is a module structure diagram of a system for multi-cluster task scheduling and monitoring according to some embodiments of this specification;
[0011] Figure 2 is an exemplary flow chart of a method for multi-cluster task scheduling and monitoring according to some embodiments of this specification;
[0012] Figure 3 is an exemplary schematic diagram of allocating a second task according to some embodiments of this specification;
[0013] Figure 4 is an exemplary schematic diagram of a load assessment model according to some embodiments of this specification;
[0014] Figure 5 is an exemplary flowchart of reallocating tasks according to some embodiments of the present specification;
[0015] Figure 6 is a schematic diagram of the structure of a system for multi-cluster task scheduling and monitoring according to some embodiments of this specification;
[0016] Figure 7 is an internal structure diagram of a system for multi-cluster task scheduling and monitoring according to some embodiments of this specification;
[0017] Figure 8 is an exemplary schematic diagram of instruction state conversion according to some embodiments of this specification;
[0018] Fig. 9 It is an exemplary schematic diagram of job state transition according to some embodiments of this specification. DETAILED DESCRIPTION
[0019] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.
[0020] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.
[0021] Unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.
[0022] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.
[0023] Figure 1 It is a module structure diagram of a system for multi-cluster task scheduling and monitoring according to some embodiments of this specification.
[0024] In some embodiments, the multi-cluster task scheduling and monitoring system can be applied to multiple fields that need to process a large amount of real-time data. For example, in a cloud computing environment, a large amount of real-time data needs to be processed and analyzed, and a multi-cluster architecture can provide better processing capabilities and scalability. The multi-cluster task scheduling and monitoring system 100 can help administrators or systems automatically select the most appropriate cluster to process tasks, while ensuring efficient allocation and execution of tasks in job components; for example, in the fields of finance, e-commerce, logistics, etc., the multi-cluster task scheduling and monitoring system 100 can ensure that when the data generation rate is high, tasks can be promptly allocated to the corresponding job components for processing, while monitoring the status of the job components, and once the task processing fails, it can quickly issue a warning and reallocation.
[0025] like Figure 1As shown, the multi-cluster task scheduling and monitoring system 100 may include a cluster registration module 110 , a component activation module 120 , a job startup module 130 , a first allocation module 140 , and a second allocation module 150 .
[0026] The registration cluster module 110 is configured to send a registration instruction to at least one to-be-registered cluster based on a user request to generate a registration cluster.
[0027] The activation component module 120 is configured to determine at least one operation component to be activated based on the inertial data volume of the registered cluster, and issue an activation instruction to at least one operation component to be activated to generate an activation operation component set corresponding to the registered cluster, where the inertial data volume is the data generation rate within a preset historical period.
[0028] The start job module 130 is configured to generate at least one start instruction based on the data volume of the first task in the first preset time period, and send at least one start instruction to the registration cluster; wherein the registration cluster includes a scheduling component and a job component, and the scheduling component is used to receive and schedule the first task to the activation job component for processing based on the start instruction.
[0029] The first allocation module 140 is configured to allocate and process the second task of the second preset time period in the activated job component set based on the load status value of the activated job component and the amount of data to be processed. The second preset time period is a time period after the first preset time period.
[0030] In some embodiments, the first allocation module 140 is further configured to: determine a reserved load set based on a second preset time period, the reserved load set including a reserved idle load of a reserved job component; treat the newly added job task as a second task; and allocate the second task based on the load status value and the amount of data to be processed of each reserved job component.
[0031] The second allocation module 150 is configured to issue a warning in response to a task processing failure based on the failure cause and the failure occurrence time, and reallocate the failed task in the activated job component set, wherein the failed task includes at least one of a failed first task and a failed second task.
[0032] In some embodiments, the second allocation module 150 is further configured to: in response to the load occupancy information corresponding to the failed task satisfying the load condition, determine a new component to be activated based on the load occupancy information; issue an activation instruction to generate a new component; based on the new component, generate an update component set; based on the historical load status value and historical failure record of each update component in the update component set, determine the job insurance value corresponding to each update component; and reallocate the failed task based on the job insurance value and the update component set.
[0033] In some embodiments, the multi-cluster task scheduling and monitoring system 100 also includes an initialization module (not shown in the figure), and the initialization module is further configured to: determine the reliability value of the activated job component based on the number of downtimes and occurrence time of the activated job component; in response to the reliability value satisfying the reliability condition, initialize the activated job component to an unactivated state.
[0034] In some embodiments, the multi-cluster task scheduling and monitoring system 100 may include a network and / or other components that connect the system to external resources. The server may obtain data and / or information related to the multi-cluster task scheduling and monitoring system 100 through the network.
[0035] In some embodiments, the multi-cluster task scheduling and monitoring system 100 may further include a terminal device. The terminal device may refer to one or more terminal devices used by a user, and the server may be integrated in the terminal device. The user may refer to an administrator or operator of the multi-cluster task scheduling and monitoring system 100.
[0036] In some embodiments, the multi-cluster task scheduling and monitoring system 100 may further include a database (not shown) for storing data, instructions and / or any other information. For example, the database may store inertial data, the first task, a task pool, etc.
[0037] In some embodiments, the database may include random access memory (RAM), read-only memory (ROM), etc. or any combination thereof. In some embodiments, the database may be integrated or included in one or more other components (e.g., servers, terminal devices, etc.) of the multi-cluster task scheduling and monitoring system 100.
[0038] Figure 6 Schematic diagram of the structure of a multi-cluster task scheduling and monitoring system according to some embodiments of this specification. Figure 6 As shown, in some embodiments, the registration cluster can be referred to as a physical cluster, and the system 100 for multi-cluster task scheduling and monitoring can also include a big data cluster. A big data cluster refers to a collection of hardware and software used to process large-scale data sets. In some embodiments, the big data cluster can include Flink, HDFS, and Yarn. Among them, Flink refers to a stream processing framework, HDFS refers to a distributed file system, and Yarn refers to a resource management system. The physical cluster can perform file management and file scheduling based on the big data cluster.
[0039] Figure 7 It is an internal structure diagram of a multi-cluster task scheduling and monitoring system according to some embodiments of this specification. Figure 7 The components in the multi-cluster task scheduling and monitoring system 100 and the interaction between the components are shown. Figure 7As shown in the figure, the server is used to issue scheduling commands, which are used to instruct the job component to execute the job. The scheduling component is used for load balancing, fault-tolerant recovery, etc. The scheduling component receives the scheduling command from the server and distributes the command to the corresponding job component. The job component is used to execute or process a specific task. After the job component completes the corresponding task, the scheduling component can report the execution status of the task to the server. The server can view the work log of the job component through the scheduling component.
[0040] For more information about the registration cluster module 110 , the activation component module 120 , the start operation module 130 , the first allocation module 140 , and the second allocation module 150 , please refer to the relevant description below.
[0041] In some embodiments, the registration cluster module 110, the activation component module 120, the start operation module 130, the first allocation module 140 and the second allocation module 150 are integrated into the server. Figure 2 The process 200 in can be considered to be executed by the server.
[0042] It should be noted that the above description of the system 100 and its modules for multi-cluster task scheduling and monitoring is only for convenience of description and does not limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules or form a subsystem connected with other modules without deviating from this principle. For example, the modules can share a database, or each module can have its own database. Such variations are all within the scope of protection of this specification.
[0043] Figure 2 is an exemplary flow chart of a method for multi-cluster task scheduling and monitoring according to some embodiments of this specification.
[0044] In some embodiments, the process 200 can be implemented based on a server of the multi-cluster task scheduling and monitoring system 100. Figure 2 As shown, process 200 includes the following steps:
[0045] Step 210: Based on the user request, a registration instruction is sent to at least one cluster to be registered, so as to generate at least one registered cluster.
[0046] A user request refers to a service request made by a user to the multi-cluster task scheduling and monitoring system 100 .
[0047] A cluster to be registered refers to a group of computing resources that are in a waiting state and waiting to process job tasks.
[0048] A registered cluster is a cluster that has been registered with the server and is in operation, capable of executing computing tasks, processing data, or providing services. The number of registered clusters can be one or more.
[0049] In some embodiments, the server may obtain a user request based on the terminal device and generate a corresponding registration cluster. Different user requests correspond to different registration clusters.
[0050] Step 220: determine at least one operating component to be activated based on the inertial data volume of the registered cluster, and issue an activation instruction to the at least one operating component to be activated to generate an activated operating component set corresponding to the registered cluster.
[0051] Inertial data volume refers to data related to the amount of data transmitted or required to be processed within a period of time. For example, the inertial data volume may include the sum of the data volume of each job within a period of time. A job is the work corresponding to processing or calculating a user's service request. A job can contain multiple tasks, and a task refers to the sum of the work that a computer needs to complete during a transaction or calculation process. For example, a job may include a data processing task, and exemplary jobs may include data storage and maintenance, data encryption and decryption, model prediction, model training, online transactions and payments, etc.
[0052] In some embodiments, the inertial data volume is a data generation rate within a preset historical period.
[0053] The preset historical period refers to a certain period of time in the past, such as the past month, a week, or any other one or more time periods in the past. The preset historical period can be a system preset value or a system default value.
[0054] In some embodiments, the preset historical period can be matched and determined based on the current time feature. The current time feature refers to the time feature including the current period, and the current period can refer to a period of time including the current moment. The time feature refers to a feature that affects the amount of inertial data and is related to time.
[0055] In some embodiments, the server may perform statistical analysis on historical data (e.g., the past week / month, etc.), determine the current time feature, and determine the preset historical period based on the current time feature. For example, by performing statistical analysis on the historical data of the past month, it can be determined that the amount of inertial data on working days is greater than the amount of inertial data on holidays, and the time feature includes working days and holidays; if the current time feature is a working day, the server may use the period of the working day as the preset historical period.
[0056] Data generation rate refers to the number of data bits transmitted or processed per unit time. The unit time can be a certain period of time as a unit, such as one second, one minute, one hour, etc.
[0057] In some embodiments, the server may determine the data generation rate within the preset historical period based on the ratio of the sum of the data volume of each job of the registered cluster within the preset historical period to the preset historical time period as the inertial data volume of the registered cluster.
[0058] The job components to be activated are job components that need to be activated to process tasks.
[0059] In some embodiments, the server can determine at least one job component to be activated in a variety of ways. For example, the server can calculate the number of job components required to process the amount of inertial data based on the processing performance of the job components (such as response time, throughput, etc.); based on the number of required job components, select the job component with the longest sleep time from the dormant job components as the job component to be activated. A dormant job component refers to a job component that is placed in a dormant state. When a dormant job component is in a dormant state, the job component is unavailable and is in an inactive state. In some embodiments, the server can determine the processing performance of the job component based on manual input or a database.
[0060] It should be noted that the job component to be activated needs to be activated before the first preset period to avoid the problem of waiting too long for activation after receiving the task. For more information about the first preset period, please refer to Figure 2 The relevant description is as follows.
[0061] An activation job component set is a set consisting of different activation job components.
[0062] The activation instruction is a series of commands used to activate the job component to be activated, which is used to change the job component to be activated from an inactive or dormant state to an active state, so that it can start processing tasks, requests or data, etc.
[0063] In some embodiments, the server may determine the set of activated job components in a variety of ways. For example, based on the determined one or more job components to be activated and the number of job components to be activated, the server may issue an activation instruction to each job component to be activated, so that the job component to be activated is transformed into an activated job component; and combine the one or more activated job components into a set of activated job components.
[0064] Step 230: Generate at least one start instruction based on the data volume of the first task in the first preset time period, and send the at least one start instruction to the registration cluster.
[0065] The first preset period refers to the period of time during which the job is waiting to be assigned. For example, the server receives the job but does not assign it immediately at intervals, and the interval is the first preset period. The first preset period can be a system preset value, a system default value, etc. In some embodiments, the first preset period is a smaller value, such as 0.1s, indicating that the server assigns the first task immediately after receiving it. In some embodiments, the server can receive the job sent by the user based on the terminal device.
[0066] In some embodiments, the preset historical period is before the first preset period. Data of the preset historical period is used as a reference or basis to support the allocation decision of the tasks in the first preset period.
[0067] The first task refers to a job or task that is received and waiting to be assigned within a first preset period of time. The server may receive multiple first tasks within the first preset period of time.
[0068] A start instruction is an instruction used to trigger the execution of a job or task. For example, a start instruction is used to notify the activation job component to start executing the first task. In some embodiments, the start instruction may include a job identifier, resource requirements, a processing order, etc. Among them, the job identifier is used to uniquely identify the job or task to be started. Resource requirements are used to specify the resources required for the job or task, such as the number of processor cores, memory size, disk space, etc. The processing order refers to the execution order of the job or task, such as the execution time, the execution order, etc.
[0069] In some embodiments, the server can generate at least one start instruction based on the data volume of the first task in the first preset period at the end of the first preset period or at a specific interval. For example, after the server assigns the first task to the registered cluster, it automatically sends the start instruction; and automatically sends the job synchronization instruction so that the activation job component synchronizes the execution result to the server after the job task is completed. Fig. 9 FIG. 1 is an exemplary schematic diagram of a job state transition according to some embodiments of this specification. Fig. 9 As shown in the figure, the job instructions for registering a cluster include pause instructions, resume instructions, kill instructions, restart instructions, start instructions, etc. For more information about unknown, please refer to Figure 2 The relevant description is as follows.
[0070] The pause command is used to temporarily suspend a running job or task and stop its execution. The resume command is used to restore a suspended job or task to a running state. The restart command is used to terminate a currently running job or task and immediately re-execute the job. The kill command is used to immediately terminate a running job and release the resources it occupies. The job synchronization command is used to provide real-time feedback to the server on the execution result of the job component on the first assigned task.
[0071] In some embodiments, the server can assign the start instruction to the activation job component in a variety of ways. For example, the server can obtain the first task assigned to each activation job component in the activation job component set by uniform distribution based on the data volume of the first task in the first preset time period, generate the corresponding start instruction based on the first task assigned to each activation job component, and send it to each activation job component. For another example, the server can assign the first task to each activation job component in the activation job component set in the order of reception, generate the corresponding start instruction based on the first task assigned to each activation job component, and send it to each activation job component. Exemplarily, there are 6 tasks in the first task, and there are 3 activation job components in the activation job component set. The server will evenly distribute the 3 tasks received at the first time, generate the corresponding first work start instruction and send it to the 3 activation job components, and the first work start instruction includes the first priority; then evenly distribute the 3 tasks received at the second time, generate the corresponding second work start instruction and send it to the 3 activation job components, and the second work start instruction includes the second priority. The priority is used to determine the processing order of the tasks. The higher the priority, the higher the priority, the higher the priority of the task. The first priority is higher than the second priority. The first time and the second time are time points in a first preset time period, and the first time is before the second time.
[0072] In some embodiments, the registration cluster includes a job component, and the job component includes an activation job component and a dormant job component, wherein the activation job component is used to process the first task based on the startup instruction.
[0073] The operating component may be a device with data acquisition, storage, calculation, analysis and / or transmission functions, such as a personal computer.
[0074] In some embodiments, the registration cluster further includes a scheduling component, and the scheduling component is used to schedule the job component.
[0075] In some embodiments, the scheduling component determines the priorities corresponding to different tasks based on the order in which the tasks are assigned and / or received. The earlier the task is assigned and / or received, the higher the priority. The server can assign high-priority tasks to job components with more processing resources (such as processor computing resources, storage resources, etc.).
[0076] In some embodiments, during the process of processing tasks, the demand for processing resources varies. The closer the processing time of a task is to the end time, the fewer processing resources the task occupies. When the time interval between the processing time of the previous task and the end time is less than the corresponding threshold, and the idle processing resources are greater than the processing resources required for the next task, the server can schedule the idle processing resources in advance to process the next task. The threshold can be a system preset value or a system default value.
[0077] In some embodiments, the server may establish a correspondence between different preset tasks and different processing resources required / occupied based on historical data.
[0078] In some embodiments, the scheduling component sets corresponding weights based on the processing resources of each job component (such as the number of processor cores, memory size, etc.), and schedules tasks for the job components through a load balancing algorithm. For example, the weight score of a job component with more processing resources is greater. A load balancing algorithm refers to an algorithm used to schedule the number of tasks received by a job component. In some embodiments, the server can determine the load weight of each job component based on the load balancing algorithm, and determine the number of first tasks that each job component needs to process based on the load weight. In some embodiments, the load balancing algorithm may include a weighted round-robin algorithm, a weighted random algorithm, or a weighted least connection method, etc.
[0079] In some embodiments, the server may determine the load score of each activated job component, and use the job component with the lowest load score as the scheduling component; calculate the weight score of each job component based on the scheduling component, and determine the number of first tasks that each job component needs to process based on the weight score. The preset algorithm refers to an algorithm or formula for calculating the load score of the job component.
[0080] For example, the server may calculate the load score based on formula (1):
[0081] Load score = (1-processor usage) × 10 + (1-memory usage) × 20 + A × 20 (1);
[0082] Wherein, A is the first coefficient, which can be determined based on the average load. For example, the server can determine whether the average load is greater than or equal to the load threshold: if greater than or equal to, A is 0; if less than, the server can calculate the difference between the load threshold and the average load, and determine A based on the ratio of the difference to the load threshold. The load threshold can be a system preset value or a system default value, etc. For example, the load threshold is 3.
[0083] The scheduling component can calculate the weight score based on formula (2):
[0084] Weight score = weight of job component / maximum weight × 30 + load score + B (2);
[0085] Wherein, B is the second coefficient, which can be determined based on whether the current job component is a scheduling component. For example, if the current job component is a scheduling component, B is 0; if the current job component is not a scheduling component, B is 20.
[0086] Among them, the current job component is the job component that needs to determine the weight score.
[0087] Figure 8 FIG. 1 is an exemplary schematic diagram of instruction state conversion according to some embodiments of this specification. Figure 8 As shown, the scheduling component forwards the server's startup instruction to the job component with a lower load score in the cluster through the load balancing mechanism; after receiving the startup instruction, the job component starts to process the task and feeds back the startup success message as the job startup result to the scheduling component. The scheduling component notifies the server to change the execution status to executing and the status of the startup instruction to successful execution; the job component can add the assigned task to the task list to continuously monitor the execution status of the task, and report the execution status to the server for display to the user; when the job component monitors that the task execution is completed (the job component is executed successfully), it will send a job end message to the scheduling component, and the scheduling component notifies the server to change the execution status to successful execution; when the number of retries of the assigned task exceeds the corresponding threshold, the job component generates a restart instruction to restart the job; when the job component fails to execute, the job component generates a warning and sends it to the scheduling component, and the scheduling component notifies the server to change the execution status to failed execution for display to the user. The threshold can be a system default value or a system preset value. When the scheduling component of the registered cluster goes offline during the execution of the task, the server cannot determine the execution status of the task and can change the status of the task to unknown.
[0088] In some embodiments, the registration cluster also includes a registration component (such as a registration center) for providing service management functions. The registration component can perform unified management of job components, including registration, deregistration, query, and execution status of job components. The scheduling component is responsible for monitoring the temporary nodes of all job components in the cluster in the registration component. When a job component goes down, the interaction between the job component and the registration component is disconnected. The registration component will delete the corresponding temporary node and feed back the notification of temporary node deletion to the scheduling component. After receiving the above notification, the scheduling component will assign the job tasks that the downed job component is responsible for to other job components; all job components will monitor the temporary nodes of the scheduling component in the registration component. When the scheduling component goes down, the job component will receive a notification from the registration component to elect the job with a lower load score as the scheduling component. For more information about execution status, see Figure 2The relevant description is as follows.
[0089] Step 240, based on the load status value of the activated job component and the amount of data to be processed, the second task of the second preset time period is allocated and processed in the activated job component set.
[0090] The load status value is used to reflect the size of the processing resources occupied when the activated job component processes the task. In some embodiments, the load status value can be an absolute value (such as the frequency or memory utilization of the processor of the current job component, disk I / O speed), or a relative value (such as the ratio of the processing resources occupied by the currently processed task to the processing resources of the registered cluster).
[0091] In some embodiments, the server can collect in real time various performance indicators of each activated job component in the registered cluster when processing tasks, such as processor utilization, memory occupancy, disk I / O, network bandwidth utilization, etc., to determine the load status value of each activated job component.
[0092] The amount of data to be processed refers to the amount of data of the first task waiting to be processed by the job component.
[0093] In some embodiments, the server may obtain in real time the task list of each activated job component in the registration cluster to determine the amount of data to be processed. The task list is a list for temporarily storing the first task to be processed.
[0094] The second preset time period refers to the time period for allocating the first task in the activated job component set.
[0095] In some embodiments, the second preset period is a time period after the first preset period.
[0096] For more information about the second preset period, please refer to Figure 3 Related description.
[0097] The second task refers to a newly added job or task that needs to be assigned within a second preset time period.
[0098] In some embodiments, the server can allocate and process the second task of the second preset time period in the activation job component set in a variety of ways. For example, for each registered cluster, the server can calculate the time it takes for each activation job component to complete the processing of the first task to be processed based on the current load state value of each activation job component in the registered cluster; determine the waiting time required to allocate the second task to the activation job component based on the processing completion time; and preferentially allocate the second task to the activation job component with the shortest waiting time.
[0099] In some embodiments, the server may determine a reserved load set based on a second preset period; use the newly added job task as the second task; and allocate the second task based on the load status value and the amount of data to be processed of each reserved job component. For more information, see Figure 3 Related description.
[0100] Step 250 , in response to the task processing failure, based on the failure cause and the time when the failure occurred, a warning is issued, and the failed task is reallocated in the set of activated job components.
[0101] Task processing failure refers to the situation where the tasks assigned to the job component cannot be successfully executed.
[0102] In some embodiments, the server can determine the execution status of each activated job component in the registration cluster for each task (such as the first task or the second task), and when the execution status is execution failure, it is determined that the task processing has failed. For more information about the execution status, please refer to Figure 5 Related description.
[0103] The failure reason refers to the reason why the job component fails to process the task. For example, the failure reason may include sudden downtime of the job component, failure to start the job, and job running errors.
[0104] In some embodiments, the server can determine the cause of failure and the failure event based on the work log of each activated job component in the registration cluster. The work log is used to record detailed information of the job component during operation, including but not limited to the operation status, key events (such as downtime, etc.), detailed information of task execution, etc. The detailed information of task execution may include the start and end time of the task, the occupied processing resources, the amount of data processed, whether it is successfully completed, etc. The work log can reflect not only the status of the task, but also the load of the job component.
[0105] The failure occurrence time refers to the starting time when the job component fails to process the task.
[0106] Warnings are used to alert you to messages related to task processing failures.
[0107] In some embodiments, the warning may include the failed task, the reason for the failure, and the time when the failure occurred.
[0108] In some embodiments, in response to a task processing failure, the server may generate a warning based on the failed task and the work log of its corresponding job component and send it to the terminal device. The warning may be prompted by displaying a pop-up window, instant message, or playing voice on the terminal device. For example, the warning may be "the job component suddenly crashed, resulting in the failure of task processing."
[0109] A failed task is a job or task that failed to execute successfully.
[0110] In some embodiments, the failed task includes at least one of a failed first task and a failed second task.
[0111] In some embodiments, the server may determine the first task or the second task when task processing fails as a failed task.
[0112] In some embodiments, the server can reallocate failed tasks in the set of activated job components in a variety of ways. For example, the server can reallocate failed tasks to the activated job components with the shortest waiting time according to the aforementioned method of allocating the second tasks.
[0113] In some embodiments, the server may monitor each registered cluster in the above manner, and in response to a task processing failure of a registered cluster, reallocate the failed task in the activated job component set.
[0114] Exemplarily, for the purpose of easy distinction, one or more activated job components can be exemplified by the first component, the second component, etc. The server can schedule the failed task in the first component to the second component. The first component is pre-assigned with one or more first tasks, such as job 101, job 102, job 103, ... Each job includes several tasks, such as task 1, task 2, task 3, task 4, ... Similarly, several jobs are also running in the second component. When a task fails to execute in the first component, it is necessary to schedule the task of the first component to run in the second component, the server implements task granularity scheduling, that is, it can schedule part of the tasks in one or more jobs running or calculated in the first component to run in the second component to improve the overall utilization efficiency of the system. For example, tasks 1 and 2 in job 101, and tasks 3 and 4 in job 102 are scheduled to run in the second component without scheduling the entire job 101 and job 102, thereby realizing flexible scheduling of tasks. The server is used to obtain the execution status of each task included in one or more jobs running on the first component, and establish a task feature library based on failed tasks; determine the scheduling strategy based on the processing performance of the second component (such as activating the job component with the shortest waiting time); select tasks matching the scheduling strategy from the task feature library to generate a list of failed tasks to be scheduled; the server is used to schedule the tasks to be scheduled from the first component to the second component based on the list of failed tasks to be scheduled. It should be noted that the above examples are only shown to facilitate the understanding of this specification, and the implementation methods of this specification are not limited in this regard.
[0115] In some embodiments, the server can determine a new component to be activated based on the load occupancy information in response to the load occupancy information corresponding to the failed task meeting the load condition; issue an activation instruction to generate the new component; generate an update component set based on the new component; determine the job insurance value corresponding to each update component based on the historical load state value and historical failure record of each update component in the update component set; reallocate the failed task based on the job insurance value and the update component set. For more information, see Figure 5 Related description.
[0116] In some embodiments, the method further includes: determining a reliability value of the activation job component based on the number of downtimes and occurrence time of the activation job component; and initializing the activation job component to an unactivated state in response to the reliability value satisfying a reliability condition.
[0117] Downtime refers to the situation where the activated job component stops running or fails and cannot process tasks normally.
[0118] The number of downtimes refers to the number of downtimes that occur to the activated job component within a period of time. The period of time can be a system preset value or determined based on manual input.
[0119] The occurrence time refers to the time when the activation job component occurs each time a downtime occurs.
[0120] In some embodiments, the server may obtain the work logs of each activation job component in the registration cluster in real time to determine the number of downtimes and occurrence times of the activation job components.
[0121] The reliability value is used to measure the ability of the activated job component to maintain the accuracy and data consistency of its processing tasks during normal operation.
[0122] In some embodiments, the reliability value may be expressed as a numerical value (eg, reliability degree, reliability value, etc.) or a level (eg, reliability level), etc.
[0123] In some embodiments, the reliability value can be determined based on the number of downtimes and the time interval. For example, the reliability value is negatively correlated with the number of downtimes and positively correlated with the time interval. The fewer the number of downtimes, the larger the mean time interval, and the larger the reliability value. The time interval refers to the time interval between two adjacent downtimes. Adjacent means that the moments corresponding to the downtime events are adjacent on the time axis.
[0124] In some embodiments, the server may determine the reliability value by determining a rule based on the number of downtimes and the time interval.
[0125] Exemplarily, the determination rule is determined based on formula (3):
[0126] Reliability value = k1 × mean of time intervals - k2 × number of downtimes (3);
[0127] Among them, k1 and k2 are coefficients greater than 0, and k1 and k2 can be determined based on experiments or experience.
[0128] In some embodiments, the server can determine the weight corresponding to each downtime based on the load rate of the activated job component in the current downtime and the time interval between the current downtime and the previous downtime; based on the weight of each downtime, the reliability value of the activated job component is determined by summing up.
[0129] The current downtime refers to the downtime event that is currently occurring or the most recent downtime event.
[0130] The last downtime refers to the most recent historical downtime event.
[0131] In some embodiments, the server may count the time interval between the current downtime and the last downtime of the activation job component based on the work logs of each activation job component in the registration cluster.
[0132] The load rate is used to measure the load of the activated job component during the current downtime.
[0133] In some embodiments, the server may determine the load rate of the activated job component at the time of the current downtime based on the usage rate, memory occupancy rate, disk I / O, and network bandwidth of each activated job component in the registered cluster. For example, the server may calculate the usage rate, memory occupancy rate, disk I / O, and network bandwidth of each activated job component in the registered cluster, and their corresponding weights, and calculate the load rate of the activated job component at the time of the current downtime by weighted summation. The weight may be determined based on experiments or experience.
[0134] In some embodiments, for each downtime, if the load rate at the time of downtime is lower, it means that the activation job component may also crash under low load conditions, and the reliability is lower, then the weight corresponding to the downtime is lower; the shorter the time interval between two adjacent downtimes, the more unstable the activation job component is, and the lower the reliability, then the weight corresponding to the downtime is lower.
[0135] In some embodiments, for each activation job component, the server may determine the reliability value of the activation job component by summing up the weight of the activation job component at each downtime.
[0136] In some embodiments of the present specification, the load rate reflects the load pressure of the server at the time of downtime, and the time interval provides information about the frequency of downtime. By considering the load rate and the time interval at the same time, the impact of downtime events on server reliability can be more comprehensively evaluated; the weight is dynamically calculated and can reflect the current status and performance of the server. As the load rate and time interval change, the weight will also be adjusted accordingly, thereby providing a more accurate reliability value.
[0137] The reliability condition is a judgment condition for evaluating whether to initialize the activation operation component. For example, the reliability condition may include that the reliability value is lower than the reliability threshold. The reliability threshold may be a system preset value, a system default value, etc.
[0138] The inactive state refers to the state of a job component before activation is completed. The job component has not yet started to provide services or process tasks.
[0139] Initialization refers to changing the activation job component from a running state to an initial, unconfigured state. For example, initialization may include stopping all tasks being executed in the activation job component, deleting records in the component's database, clearing cache or temporary files, etc.
[0140] In some embodiments, for each activation job component, the server can determine whether the reliability value of the activation job component is lower than the reliability threshold. If so, the activation job component is initialized; and the unprocessed first task of the activation job component is treated as a failed task and reallocated according to the aforementioned method of allocating the second task.
[0141] In some embodiments, the server may reallocate the unprocessed first task of the activation job component by adding an activation job component and calculating the job insurance value. For more information, see Figure 3 Related description.
[0142] In some embodiments of the present specification, by counting the number of downtimes and occurrence times of the activation job component in real time or regularly, the current state of the activation job component can be quickly evaluated; by monitoring the number of downtimes and occurrence times, the potential failure risk of the activation job component can be identified. If the number of downtimes is frequent or the occurrence time is concentrated in a certain time period, this may mean that there are some unstable factors or configuration problems in the activation job component. Initializing the activation job component to an inactive state in a timely manner can avoid further risk expansion.
[0143] In some embodiments of the present specification, determining the activation job component based on the amount of inertial data can ensure that the task is processed at the fastest speed and reduce the waiting time of the task. Because the activation job component generates heat when processing the task, which affects the actual performance and loss, the activation job component with the longest sleep time has dissipated heat and cooled down, and can process the task in the best state. Task allocation based on the load status value of the activation job component and the amount of data to be processed can further ensure the balanced distribution of tasks and avoid overloading of the activation job component or idle resources. Through task scheduling and monitoring, the activation job component can be automatically selected and tasks can be allocated based on real-time data, and warnings can be automatically issued and tasks can be reallocated when task processing fails, thereby improving management efficiency, reducing human errors, and making the system more flexible and responsive.
[0144] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.
[0145] Figure 3 is an exemplary schematic diagram of allocating a second task according to some embodiments of the present specification.
[0146] In some embodiments, Figure 3 As shown, the server can determine a reserved load set 370 based on a second preset time period 360, the reserved load set including a reserved idle load of a reserved job component; take a newly added job task as a second task 390; and allocate the second task 390 based on a load status value 371 and an amount of data to be processed 380 of each reserved job component.
[0147] For more information about load status values, the amount of data to be processed, and the second task, see Figure 2 Related description.
[0148] In some embodiments, the server may determine the second preset time period in a variety of ways. For example, the server may determine the second preset time period based on the time between two adjacent first preset time periods.
[0149] In some embodiments, Figure 3 As shown, the second preset period 360 is the estimated allocation period 350 of the first task, and the estimated allocation period 350 is determined based on the total number 331 of the first tasks.
[0150] The estimated allocation period is the estimated time period for allocating the first task. For example, the estimated allocation period may include an estimated allocation start point, an estimated allocation end point, an allocation duration, etc. For more information about the estimated allocation start point, the estimated allocation end point, and the allocation duration, please refer to Figure 3The relevant description is as follows.
[0151] In some embodiments, the first preset period is a predetermined value, and the server can determine the estimated allocation starting point of the estimated allocation period based on the first preset period. For example, the server can determine the estimated allocation starting point based on the end time of the first preset period, and the server can collect historical data to establish a preset table of allocation durations required for different numbers of tasks to be allocated to different activated job components, and determine the allocation duration by looking up the table based on the total number of first tasks received within the first preset period: determine the estimated allocation period based on the estimated allocation starting point and the allocation duration.
[0152] The estimated allocation start point and the estimated allocation end point refer to the estimated start time and end time of allocating the first task.
[0153] The allocation duration refers to the duration required to allocate all the first tasks, that is, the time length between the estimated allocation start point and the estimated allocation end point.
[0154] For more information about the estimated allocation starting point and allocation duration, see Figure 3 The relevant description is as follows.
[0155] In some embodiments of the present specification, by dynamically adjusting the estimated allocation period according to the number of tasks, task requirements and processing resources can be better matched. If the total number of first tasks is large, the allocation time can be extended to fully utilize the processing resources; if the number of tasks is small, the allocation time can be shortened to avoid wasting resources, which helps to maximize the utilization of resources.
[0156] In some embodiments, Figure 3 As shown, the server can count the data volume distribution 321 of the first task in the task pool 310; based on the data volume distribution 321 and the inertial data volume 322, predict the time when the total number of first tasks 331 and the total data volume 332 of the first tasks meet the allocation condition 330; use the time that meets the allocation condition 330 as the estimated allocation starting point 341, and determine the allocation duration 342 based on the estimated allocation starting point 341; determine the estimated allocation time period 350 based on the estimated allocation starting point 341 and the allocation duration 342.
[0157] For more information about inertial data, see Figure 2 Related description.
[0158] A task pool is a component or container for storing jobs or tasks that have been received but not yet assigned.
[0159] The data volume distribution is used to reflect the distribution of the data volume of the first task over time. For example, the data volume distribution includes statistical values such as the mean, mode, and median of the data volume of all first tasks in different time periods.
[0160] In some embodiments, the server may perform statistical analysis on the data received within the first preset time period based on the task pool to determine the data volume distribution.
[0161] The total number of first tasks refers to the total number of first tasks that need to be processed or executed within the first preset time period.
[0162] The total data volume of the first task refers to the total amount of data that needs to be processed or generated by all the first tasks.
[0163] The allocation condition is an evaluation condition for determining the estimated allocation starting point. For example, the allocation condition may include that the total number of the first task is greater than a first threshold, and / or that the total amount of data of the first task is greater than a second threshold. The first threshold and the second threshold may be determined based on experiments or experience.
[0164] It should be noted that, under the same total data volume, the greater the total number of first tasks, the smaller the data volume of each first task, the more serious the data fragmentation, and the slower the job component processes it, which will affect the waiting time for processing.
[0165] In some embodiments, the server may determine the first threshold in a variety of ways. For example, the server may obtain the retention time corresponding to each of the total number of different historical first tasks in the task pool; and determine the first threshold based on the proportion of pause instructions, kill instructions, and termination instructions corresponding to different retention times. Exemplarily, the higher the proportion, the lower the first threshold.
[0166] The retention time refers to the length of time that the first task waits for assignment in the task pool.
[0167] When there are too many first tasks, the first tasks remain in the business pool for a long time because they need to wait for allocation. Users may issue pause commands, kill commands, or terminate commands when the job is abnormal (such as a slow progress bar or no response). The higher the proportion of pause commands, kill commands, or terminate commands, the longer the job remains due to the excessive number of first tasks, which leads to a higher possibility of job abnormality. A lower first threshold needs to be set.
[0168] In some embodiments, the server can determine the second threshold in a variety of ways. For example, after allocating the historical first tasks with different total data volumes in the task pool, the server can count the load status values of the activated job components; select the job component whose load status value is closest to and does not exceed the job insurance value of the activated job component, and use the total data volume corresponding to it as the second threshold. The job insurance value is the maximum load value that ensures the stable operation of the activated job component.
[0169] For more information about job insurance values, see Figure 5Related description.
[0170] In some embodiments, the allocation condition is related to a historical reliability value of the activation job component.
[0171] For more information about activating job components, see Figure 2 Related description.
[0172] The historical reliability value of the activated job component refers to the reliability value of the activated job component in the past period of time.
[0173] For more information about reliability values, see Figure 2 Related description.
[0174] In some embodiments, the higher the historical reliability value, the more stable the operation of the activated job component is, and the more jobs or tasks can be processed. The server can calculate the average of the historical reliability values of all activated job components within a preset historical period; and adjust at least one of the first threshold and the second threshold according to the average of the historical reliability values. For example, different means of historical reliability values correspond to different first thresholds and second thresholds; the larger the mean of the historical reliability values, the larger the first threshold and the second threshold.
[0175] In some embodiments of the present specification, the historical reliability value of the activation job component reflects the stability and performance of the activation job component over the past period of time. By determining the allocation conditions based on the historical reliability value, the start time of assigning tasks to the activation job component can be accurately determined, which helps to reduce the risk of task failure and data loss due to failure of the activation job component and enhance the reliability and stability of the system.
[0176] In some embodiments, the server can determine the estimated allocation period based on the estimated allocation starting point and allocation duration in a variety of ways. For example, the server can determine the statistical value (such as the mean) of the data volume of the first task in the same historical period based on the data volume distribution of the first task; predict the data volume of a single first task received in the first preset period based on the statistical value of the data volume; determine the total number of first tasks received per unit time based on the inertial data volume and the data volume of a single first task. For example, the server can determine the total number of first tasks received per unit time based on the ratio of the inertial data volume to the data volume of a single first task. Among them, the data volume of a single first task refers to the size of a single first task.
[0177] In some embodiments, the server can calculate the time when the total number of first tasks reaches the allocation conditions based on the total number of first tasks received per unit time; calculate the time when the total data volume of the first tasks reaches the allocation conditions based on the inertial data volume; when at least one of the total number of first tasks and the total data volume of the first tasks meets the allocation conditions, the time when the allocation conditions are met is determined as the estimated allocation starting point.
[0178] In some embodiments, the server may determine the allocation duration in a manner similar to a preset table. The server may calculate the estimated allocation period based on the estimated allocation start point and the allocation duration. For example, if the estimated allocation start point is 13:20:00 and the allocation duration is 10 seconds, then the estimated allocation period is 13:20:00-13:20:10.
[0179] In some embodiments of the present specification, the server will first place the first tasks that arrive within the first preset time period into the task pool. The task pool will set an upper limit on the total number of first tasks and the total data volume to accommodate, so as to avoid an excessive accumulation of the total number of first tasks, resulting in a large number of job tasks waiting for allocation and waiting for processing for too long (the more the number, the longer the allocation time); at the same time, to avoid an excessive total data volume of the first tasks, resulting in an excessive processing load on the subsequent activation of the job component, thereby increasing the probability of job failure (excessive job load may cause problems such as job component downtime).
[0180] The reserved load set is a set that reflects the distribution of idle load values reserved by different activated job components in the registered cluster. For example, the reserved load set is a set consisting of reserved idle loads of multiple activated job components. In some embodiments, the reserved load set can be represented by a vector form {A, B, C, ...}, and the A, B, and C elements in the sequence respectively represent the distribution of the reserved idle loads of different activated job components.
[0181] In some embodiments, the reserved load set includes a reserved idle load of a reserved job component. The reserved job component is an activated job component that reserves part of the idle load capacity in advance. The reserved idle load refers to the reserved, unused load capacity or processing resources of the reserved job component. For example, the reserved idle load may include a reserved unused processor core, unoccupied memory, etc.
[0182] In some embodiments, the reserved load set can be determined in a variety of ways. For example, the reserved load set can be determined by a server. For example, a job component with higher processing resources (such as processor computing resources, data storage resources, etc.) can be determined as a reserved job component, and the reserved load set is determined based on the reserved job component and its corresponding reserved idle load. Different reserved job components correspond to different reserved idle loads, and the corresponding relationship can be determined based on historical data. In some embodiments, the reserved load set can be manually preset.
[0183] In some embodiments, the server may determine the reserved idle load for one or more activated job components according to a preset ratio based on the duration of the second preset period to form a reserved load set. The longer the second preset period, the more reserved job components, and the higher the ratio of the reserved idle load to the total processing resources of the entire job component. The longer the second preset period, the more second tasks may be received, and more reserved idle loads can be used to allocate more second tasks. The preset ratio can be a system default value, a system preset value, etc., such as a preset ratio of 20%.
[0184] It should be noted that when other computing resources are available, the first task does not occupy the reserved idle load. When other computing resources are not available, the server can schedule the first task based on the reserved idle load.
[0185] For partially activated job components, a portion of idle load can be reserved in advance to handle urgent job tasks, while avoiding the problem of errors and system crashes caused by the activated job components being in a full-load state for a long time.
[0186] In some embodiments, the server can determine the reserved idle loads of different reserved job components through a load evaluation model based on the second preset period, the data volume distribution of the first task, the number of failures in the first preset period, and the job insurance value of the activated job component. For more information, see Figure 4 Related description.
[0187] The newly added job task refers to a job or task newly added within the second preset time period. In some embodiments, the server can obtain the newly added job task issued by the user within the second preset time period based on the terminal device as the second task.
[0188] In some embodiments, the server may query the load status value of the reserved job component in the reserved load set when acquiring one or more second tasks; and preferentially allocate the second task to the reserved job component in the reserved load set with the smallest load status value and the least amount of data to be processed. The processing order of the second task is after the processing order of the first job task allocated last time.
[0189] The activated job component with the smallest load status value is selected to process the second task to avoid overloading of the job component.
[0190] In some embodiments of the present specification, allocating the second task based on the load status value and the amount of data to be processed of each reserved job component can ensure that the task is allocated to a job component with a lighter load and sufficient processing power, which helps to balance the load of the activated job components and improve overall processing efficiency.
[0191] Figure 4 is an exemplary schematic diagram of a load assessment model according to some embodiments of the present specification.
[0192] In some embodiments, Figure 4 As shown, the server can process the second preset time period 360, the data volume distribution 412 of the first task, the number of failures 413 in the first preset time period, and the job insurance value 414 of the activated job component based on the load assessment model 430 to determine the reserved idle load 440 of different reserved job components; based on the reserved idle load 440 of different reserved job components, determine the reserved load set 370.
[0193] For more information about the first task, the first preset period, and the activation of the job component, see Figure 2 Related description.
[0194] For more information about data volume distribution, see Figure 3 Related description.
[0195] For more information about job insurance values, see Figure 5 Related description.
[0196] The load assessment model is a model used to determine the reserved idle load of each activation job component.
[0197] In some embodiments, the load assessment model may be a machine learning model. For example, the load assessment model is a machine learning model of a custom structure described below. The load assessment model may also be a machine learning model of other structures, including one or a combination of a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), etc.
[0198] In some embodiments, the inputs of the load assessment model may include the second preset time period, the data volume distribution of the first task, the number of failures in the first preset time period, and the job insurance value of the activated job component. The inputs of the load assessment model may include the reserved idle load of the activated job component.
[0199] The number of failures in the first preset period refers to the number of jobs or tasks that failed to be successfully executed within the first preset period. For more information about job insurance value, please refer to Figure 5 Related description.
[0200] It should be noted that the data volume distribution of the first task (such as mean, median, etc.) affects the size of the load required when the activated job component processes the first task, thereby affecting the size of the reserved idle load; jobs or tasks that fail to execute successfully within the first preset period may occupy the idle time of other job components, thereby requiring a larger reserved idle load; considering the job insurance value to determine the reserved load value can ensure the stability of the activated job component. It should be noted that the allocation of jobs is dynamically allocated in batches. During the allocated time period of the first task of the current batch, the first task of the previous batch may still be in the process of execution. Therefore, the server can count the number of failed tasks within the first preset period and determine the number of failures in the first preset period.
[0201] In some embodiments, the server may execute a load assessment model based on each activated job component, and output the reserved space load of the corresponding activated job component; and form a reserved load set based on the reserved idle loads of multiple activated job components.
[0202] In some embodiments, the load assessment model can be trained based on a large number of labeled training samples through various feasible methods. For example, parameters can be updated based on the gradient descent method. An exemplary training process includes: inputting multiple labeled training samples into the initial load assessment model, constructing a loss function through the labels and the results of the initial load assessment model, and iteratively updating the parameters of the initial load assessment model based on the loss function through gradient descent or other methods. When the preset conditions are met, the model training is completed and a trained load assessment model is obtained. Among them, the preset conditions can be that the loss function converges, the number of iterations reaches a threshold, etc.
[0203] In some embodiments, the training sample may include multiple training samples, and the training sample includes at least a sample second preset period, a sample data volume distribution of a sample task, a sample failure number of a sample preset period, and a sample job insurance value of a sample activation job component. The training sample may be obtained based on historical data.
[0204] In some embodiments, the label may include a reserved idle load corresponding to the sample activation job component. The label may be obtained by a processor or manual annotation. For example, it may be determined manually based on the load value occupied by the activation job component in processing the second task within a second preset period of history, for example, by multiplying a specified coefficient (e.g., 1.5) by the load value occupied by the second task as the label.
[0205] In some embodiments of the present specification, the reserved idle load corresponding to the job component to be activated can be predicted efficiently and accurately through the load assessment model, which can achieve better results than direct experience-based determination, and is conducive to the subsequent determination of the most appropriate reserved load set.
[0206] Figure 5 is an exemplary flowchart of reallocating tasks according to some embodiments of the present specification.
[0207] In some embodiments, process 500 may be implemented based on a server of a multi-cluster task scheduling and monitoring system. Figure 5 As shown, process 500 includes the following steps:
[0208] Step 510, in response to the load occupancy information corresponding to the failed task satisfying the load condition, determining a new component to be activated based on the load occupancy information;
[0209] For more information about failed tasks, see Figure 2 Related description.
[0210] The load occupancy information refers to the load information actually occupied by the failed task during processing and running. For example, the load occupancy information includes the occupancy of processing resources (such as the utilization rate of the processor) and the occupancy of memory (the amount of memory used) when processing the failed task.
[0211] In some embodiments, different moments correspond to different load occupancy information, and the load occupancy information may also include the load occupancy information of different periods during the processing of each failed task. The load occupancy information of each period is the average value of the load occupancy information of all moments in the corresponding period.
[0212] In some embodiments, the server can determine the load occupancy information by determining the resource occupancy (e.g., processor usage), memory occupancy (amount of memory used), and other information of the activated job components when processing failed tasks based on the work logs of each activated job component in the registration cluster.
[0213] The load condition is a judgment condition for evaluating whether a new job component to be activated needs to be added. For example, the load condition may include that the load occupancy information of the failed task is greater than a limit threshold. The limit threshold may be a system default value, a system preset value, etc.
[0214] In some embodiments, the server may obtain satisfaction at different times; calculate the satisfaction as the sum of the load occupancy information of all failed tasks corresponding to the dissatisfied moments; and use the minimum value of the sum of the load occupancy information at different times as the limiting threshold. Response time refers to the time it takes for the server to respond to a failed task. Satisfaction is used to reflect the user's satisfaction with the response time of failed tasks at different times. For example, satisfaction can be expressed as a satisfaction level (e.g., satisfied, average, dissatisfied), etc. Response time refers to the time it takes for the server to reallocate a failed task.
[0215] In some embodiments, satisfaction can be determined based on response time. For example, the server can preset a correspondence between intervals of different response times and satisfaction levels, and determine the satisfaction level corresponding to a certain moment based on the response time at that moment. The correspondence can be determined based on prior knowledge or historical data.
[0216] In some embodiments, the server may determine the load condition based on the total data volume of the second task and the reserved load set.
[0217] The total data volume of the second task refers to the total amount of data that needs to be processed or generated by all the second tasks.
[0218] In some embodiments, the server may perform statistical analysis on the second tasks received within the second preset time period in real time to determine the total data volume of the second tasks.
[0219] In some embodiments, the load condition may also include that the sum of the load occupancy information of the failed task and the processing resources of the second task is greater than the reserved load set. The server may determine the processing resources required to complete the execution of all second tasks based on the total data volume of the second task as the processing resources of the second task, and judge whether the sum of the load occupancy information of the failed task and the processing resources of the second task is greater than the reserved load set: if greater, it means that the reserved load set cannot process the failed task, and it is determined that a new job component to be activated needs to be added.
[0220] In some embodiments of the present specification, by considering the total data volume of the second task and the reserved load set, the system can take into account current and possible future load requirements when allocating tasks. When the reserved load set cannot handle failed tasks and the second task, it helps to add new activated job components in a timely manner and improve the response rate of the system.
[0221] Newly added components to be activated refer to newly added job components to be activated.
[0222] For more information about the job components to be activated, see Figure 2 Related description.
[0223] In some embodiments, the server can obtain the maximum value of the load occupancy information of failed tasks in multiple time periods; calculate the sum of the maximum values of each failed task among multiple failed tasks; compare the sum of the maximum values with the load conditions, and when the sum of the maximum values is greater than the limit threshold, determine a new component to be activated based on the load occupancy information.
[0224] When a task fails, it needs to be processed again. If the original activated job component is used to process it, it needs to wait for restart, which may affect the processing progress of the job task. Therefore, when the actual load occupancy information of the failed job task reaches the preset load condition, the job component to be activated can be added and activated to process these failed job tasks, reduce the job execution time and reduce the impact on other jobs.
[0225] In some embodiments, the server can determine the number of new components based on the load occupancy information of each failed task and the processing resources of the activated job component set; based on the number of new components, one or more job components with the longest dormant time are selected from the dormant job components as new components.
[0226] In some embodiments, the server may obtain the maximum value of the load occupancy information of failed tasks in different time periods, calculate the sum of the maximum values of each failed task in multiple failed tasks as the total load occupancy information; and determine the number of newly added components by preset rules based on the total load occupancy information and the processing resources of the newly added components. Exemplarily, the preset rule is: based on the number of newly added components, the total processing resources of all newly added components are obtained, and the total processing resources of the newly added components are greater than the product of the total load occupancy information and the safety factor. The safety factor may be a value greater than 1.
[0227] Since a job or task fails during processing, the actual execution of the failed task may require more load occupancy information. Setting a safety factor can reduce the possibility of another job failure.
[0228] In some embodiments, the safety factor can be determined based on the proportion of sudden downtime of job components in the failure reasons. The higher the proportion of sudden downtime of job components, the lower the performance of the activated job component set, and the safety factor needs to be increased, and more performance redundancy needs to be configured to ensure stable execution of the job.
[0229] Step 520: Issue an activation instruction to generate a new component.
[0230] In some embodiments, the server may issue an activation instruction to the newly added component to be activated to generate the newly added component.
[0231] For more information about activation instructions, see Figure 2 Related description.
[0232] Step 530: Generate an updated component set based on the newly added components.
[0233] The update component set refers to the updated activation operation component set. The update component set may include multiple update components, and the update component refers to one of the newly added activation operation components or the newly added components.
[0234] In some embodiments, the server may generate an update component set based on the newly added components in a variety of ways. For example, the server may select the longest dormant job component from the dormant job components based on the number of newly added components as the newly added component to be activated, and generate an activation instruction to activate the corresponding newly added component to be activated to obtain the newly added component; and add the activated newly added component to the activation job component set to obtain the update component set.
[0235] Step 540: Determine the job insurance value corresponding to each update component based on the historical load state value and historical failure record of each update component in the update component set.
[0236] The job insurance value is used to represent the maximum amount of tasks or the maximum load that the update component can stably process within a specific period of time.
[0237] The historical load status value is used to reflect the distribution of the load status value of the update component in the past period of time. For more information about the load status value, see Figure 2 Related description.
[0238] The historical failure record is used to record the task processing failures of each job component in the update component in the past period of time.
[0239] In some embodiments, different historical moments correspond to different historical load state values and different historical failure records.
[0240] In some embodiments, the server may obtain historical work logs of each job component in different update components to determine historical load status values and historical failure records of the update components.
[0241] In some embodiments, the server may calculate the job failure rate corresponding to the historical load state value of the update component at different historical moments based on the historical load state value and the historical failure record; and use the load state value whose job failure rate is less than and closest to the preset state value as the job insurance value of the update component. The preset state value reflects the stability requirement of the update component. The preset state value may be determined based on manual input or preset by a technician or user. The job failure rate refers to the proportion of failed tasks of the update component at a certain moment to all assigned tasks.
[0242] Step 550 : reallocate the failed task based on the job insurance value and the updated component set.
[0243] In some embodiments, the server may re-assign failed tasks to the update component with the largest difference between the current load status value and the job insurance value, so as to ensure that each update component is assigned the least failed tasks and reduce the waiting time required for the failed tasks to be re-executed. The current load status value refers to the load status value corresponding to the update component at the time when the failed tasks need to be re-assigned.
[0244] It should be noted that in order to ensure that each updated component is reallocated to as few jobs or tasks as possible, in addition to the job components in the newly added components, the activated job components with lower current load status values may also be allocated to failed tasks.
[0245] In some embodiments, the server can determine whether the activation job component needs to be replaced based on the execution status and the load status value of the activation job component; in response, the first task and the second task of the activation job component that needs to be replaced are reallocated based on the update component set.
[0246] The execution status refers to the execution progress of the task by the job component. For example, the execution status may include the different stages and conditions of the task from the start of processing to completion. Exemplarily, the execution status includes queuing and queuing order (e.g., 3rd order in the queue), executing and execution progress (e.g., 30% executed), execution completed, execution failed, etc.
[0247] In some embodiments, the execution status includes at least one of an execution status of the first task, an execution status of the second task, and the like.
[0248] In some embodiments, the server can obtain the work log of each activation job component in the registration cluster and determine the execution status of each task (such as the first task or the second task) of each activation job component in the registration cluster. For more information about the work log, see Figure 2 Related description.
[0249] In some embodiments, the server can determine whether the activation job component needs to be replaced based on the execution status and the load status value of the activation job component in a variety of ways. For example, the server can determine that the activation job component needs to be replaced for a task that failed to execute (such as the first task or the second task), and reallocate it based on the update component set. For another example, the server can calculate a first estimated execution time for a task in the queue (such as the first task or the second task) based on the current processing speed and the amount of data of the task before the task; and calculate the second estimated execution time of replacing the task to the update component with the smallest load status value; if the second estimated execution time is less than the first estimated execution time, it is determined that the activation job component needs to be replaced, and the task is reallocated to the update component with the smallest load status value.
[0250] In some embodiments, the server may use the above method to judge each task in the queue in turn to determine whether reallocation is required.
[0251] In some embodiments of the present specification, after a new activation job component is added, the newly added activation job component will have a lot of idle processing resources and a relatively small load status value. Therefore, job tasks that are queued later and have not yet been executed can be assigned to the newly added activation job component, which helps to reduce the queuing time and avoid waste of resources.
[0252] In some embodiments, the server may sample the current task at a preset collection frequency to obtain process data at multiple discrete time points; fit the process data at multiple discrete time points to obtain the execution speed; and reallocate the current task based on the execution speed.
[0253] The current task refers to the job or task being executed at the current moment. For example, the current task may include the current first task, the current second task, etc.
[0254] The preset collection frequency refers to the preset time interval or frequency for regularly collecting the status information of the executing task. By predicting the sampling frequency, each sampling can obtain the process data of the current task, which is helpful to understand the real-time progress of the task.
[0255] In some embodiments, the preset acquisition frequency may be a system preset value, a system default value, or determined based on manual input.
[0256] In some embodiments, the preset acquisition frequency is determined based on the task replacement frequency of other threads of the activation job component.
[0257] Other threads are threads that execute other job tasks. Each activated job component can execute multiple tasks at the same time, and the execution of each task is a thread.
[0258] Task change frequency refers to the number of times or rate at which a thread executes a task within a given period of time. It is used to measure the workload and efficiency of a thread.
[0259] In some embodiments, the preset acquisition frequency is positively correlated with the task change frequency of other threads of the activated job component. When a task of a thread is changed, the processing performance of other threads may be affected. Therefore, the higher the task change frequency of other threads, the faster the preset acquisition frequency.
[0260] In some embodiments of this specification, the preset acquisition frequency is determined by the task change frequency of other threads, and the system can monitor the performance changes of the activated job component in more real time. When the task execution frequency of other threads increases, it means that the workload of the activated job component increases. At this time, more frequent acquisition can more accurately capture performance fluctuations and discover potential problems in a timely manner.
[0261] A discrete time point refers to a time point at which a sample is taken from the task processing process to obtain process data.
[0262] Process data refers to the data processing progress of the current task.
[0263] Execution speed refers to the speed at which the job components are activated to process the job or task.
[0264] In some embodiments, the server may establish a coordinate system with the time axis as the horizontal axis and the process data as the vertical axis, and fit the process data corresponding to each discrete time point through a fitting algorithm to determine the expression of the curve; the execution speed is determined based on the expression of the curve. For example, the slope of the curve at each time point may be calculated as the execution speed. The execution speed may reflect the execution speed of tasks in the past time period and the future time period. The fitting algorithm may include the least squares method, the Levenberg-Marquardt algorithm, the genetic algorithm, etc.
[0265] In some embodiments, the server may calculate the first completion time of the current task at the execution speed based on the execution speed of the activated job component; calculate the second completion time of re-executing the current task using the newly added component based on the processing performance of the newly added component; if the second completion time is less than the first completion time, reallocate the current task based on the newly added component. For example, the current task is allocated to the newly added component corresponding to the smallest second completion time. The processing performance may include the total data volume of the task completed by the newly added component per unit time.
[0266] In some embodiments, the server can calculate the second completion time of re-executing the current task using the newly added component based on the processing performance of the newly added component in a variety of ways. For example, the server can obtain historical process data of tasks that are the same or similar to the current task from the historical data of the newly added component, and determine the completion time of the historical process data as the second completion time. For another example, the server can use a variety of methods such as manual analysis, theoretical calculation and / or modeling to determine the second completion time of re-executing the current task using the newly added component based on the processing performance of the newly added component, the complexity of the current task, etc.
[0267] In some embodiments of the present specification, reallocating the current task by execution speed can not only speed up the completion time of the current task, but also release the computing resources of the original activated job component, speeding up the completion time of other job tasks.
[0268] In some embodiments of the present specification, when an activation job component or node fails or a task fails, a new activation job component can be quickly activated to replace or take over the failed task, thereby ensuring high availability of the service and reducing system downtime; through dynamic management of the activation job component, the load distribution of the activation job component can be adjusted in real time according to the load occupancy information, thereby avoiding overloading of a single node or activation job component and improving the response speed and stability of the overall system; tasks are dynamically allocated according to the load and performance of the activation job component, thereby ensuring efficient utilization of resources and avoiding waste of resources, allowing the system to respond to various emergencies more flexibly, whether it is a change in business demand or a hardware failure, it can be responded to by dynamically adjusting the activation job component, thereby increasing the flexibility and reliability of the system.
[0269] One or more embodiments of the present specification also provide a multi-cluster task scheduling and monitoring apparatus, including a processing device, wherein the processing device is used to execute a multi-cluster task scheduling and monitoring method as described in any of the above embodiments.
[0270] One or more embodiments of this specification also provide a computer-readable medium, which stores computer instructions. When a computer reads the computer instructions in the medium, the computer runs a multi-cluster task scheduling and monitoring method as described in any of the above embodiments.
[0271] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.
[0272] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.
[0273] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.
[0274] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.
[0275] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining the digits. Although the numerical domains and parameters used to confirm the breadth of the range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values is as accurate as possible within the feasible range.
[0276] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification is hereby incorporated by reference in its entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.
[0277] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.
Claims
1. A method for multi-cluster task scheduling and monitoring, characterized in that: The method is executed by a server, and includes: Based on the user request, sending a registration instruction to at least one cluster to be registered to generate a registration cluster; Based on the inertial data volume of the registered cluster, determining at least one operation component to be activated, and issuing an activation instruction to the at least one operation component to be activated to generate an activated operation component set corresponding to the registered cluster, wherein the inertial data volume is a data generation rate within a preset historical period; Generate at least one start instruction based on the data volume of the first task in the first preset time period, and send at least one of the start instructions to the registration cluster; wherein the registration cluster includes a scheduling component and a job component, and the scheduling component is used to receive and schedule the first task to the activation job component for processing based on the start instruction; Based on the load state value of the activated job component and the amount of data to be processed, the second task of the second preset time period is allocated and processed in the activated job component set, where the second preset time period is a time period after the first preset time period; And in response to a task processing failure, based on the failure cause and the time when the failure occurred, a warning is issued, and the failed task is reallocated in the activated job component set, wherein the failed task includes at least one of a failed first task and a failed second task.
2. The method according to claim 1, characterized in that The allocating the second task for the second preset time period in the set of activated job components based on the load state value of the activated job component and the amount of data to be processed includes: Based on the second preset time period, determining a reserved load set, the reserved load set including a reserved idle load of a reserved job component; The newly added task is used as the second task; The second task is allocated based on the load status value of each of the reserved job components and the amount of data to be processed.
3. The method according to claim 1, characterized in that The reallocating the failed task comprises: In response to the load occupancy information corresponding to the failed task satisfying the load condition, determining a new component to be activated based on the load occupancy information; Issue activation instructions to generate new components; Based on the newly added components, generate an updated component set; Determine the job insurance value corresponding to each update component based on the historical load state value and historical failure record of each update component in the update component set; The failed task is reallocated based on the job insurance value and the set of updated components.
4. The method according to claim 1, characterized in that The method further comprises: Determining a reliability value of the activation operation component based on the number of downtimes and occurrence time of the activation operation component; In response to the reliability value satisfying a reliability condition, the activation operation component is initialized to an inactive state.
5. A multi-cluster task scheduling and monitoring system, characterized in that: The system comprises: A registration cluster module is configured to send a registration instruction to at least one cluster to be registered based on a user request to generate a registration cluster; an activation component module, configured to determine at least one operation component to be activated based on the inertial data volume of the registered cluster, and issue an activation instruction to the at least one operation component to be activated to generate an activation operation component set corresponding to the registered cluster, wherein the inertial data volume is a data generation rate within a preset historical period; A startup job module is configured to generate at least one startup instruction based on the data volume of the first task in the first preset time period, and send at least one of the startup instructions to the registration cluster; wherein the registration cluster includes a scheduling component and a job component, and the scheduling component is used to receive and schedule the first task to the activation job component for processing based on the startup instruction; and a first allocation module configured to allocate and process a second task of a second preset time period in the set of activated job components based on the load state value of the activated job components and the amount of data to be processed, wherein the second preset time period is a time period after the first preset time period; The second allocation module is configured to issue a warning in response to a task processing failure based on the failure cause and the time when the failure occurred, and reallocate the failed tasks in the activated job component set, wherein the failed tasks include at least one of the failed first task and the failed second task.
6. The system according to claim 5, characterized in that The first allocation module is further configured to: Based on the second preset time period, determining a reserved load set, the reserved load set including a reserved idle load of a reserved job component; The newly added task is used as the second task; The second task is allocated based on the load status value of each of the reserved job components and the amount of data to be processed.
7. The system according to claim 5, characterized in that The second allocation module is further configured to: In response to the load occupancy information corresponding to the failed task satisfying the load condition, determining a new component to be activated based on the load occupancy information; Issue activation instructions to generate new components; Based on the newly added components, generate an updated component set; Determine the job insurance value corresponding to each update component based on the historical load state value and historical failure record of each update component in the update component set; The failed task is reallocated based on the job insurance value and the set of updated components.
8. The system according to claim 7, characterized in that The system further includes an initialization module, which is further configured to: Determining a reliability value of the activation operation component based on the number of downtimes and occurrence time of the activation operation component; In response to the reliability value satisfying a reliability condition, the activation operation component is initialized to an inactive state.
9. A device for multi-cluster task scheduling and monitoring, characterized in that: The apparatus comprises at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is used to execute at least part of the computer instructions to implement the multi-cluster task scheduling and monitoring method according to any one of claims 1 to 4.
10. A computer-readable medium, characterized in that The medium stores computer instructions. When the computer reads the computer instructions in the medium, the computer executes the multi-cluster task scheduling and monitoring method as described in any one of claims 1 to 4.
