Dispatching system stability detection method, system, electronic device and storage medium

Abstract

The present application provides a scheduling system stability detection method, system, electronic device, and storage medium. The method comprises: obtaining task execution data and system scheduling data corresponding to the scheduling system; wherein the task execution data is used to indicate the operating status of the scheduling system when executing a task, and the system scheduling data is used to indicate completed tasks; calculating system stability information based on the task execution data and the system scheduling data; if the system stability information does not meet the requirements, calling an alarm module to send a prompt message of the corresponding level. In this way, the overall operation of the system can be statistically analyzed to improve the accuracy and comprehensiveness of the detection. Moreover, whether the system has an abnormality is determined based on the system's own stability information, thereby improving flexibility and the timeliness of the detection.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a scheduling system stability detection method, system, electronic device and storage medium. Background Art

[0002] As data becomes increasingly important as a factor of production, companies are processing and integrating their data to form big data businesses. Big data platform technology frameworks support a wide variety of development languages, and developers come from diverse backgrounds. This results in a large number of different types of tasks running on big data platforms, often with complex dependencies. The job scheduling system, a core component of big data development platforms, is designed to address diverse scenarios and complex content, accommodating the complex dependencies between jobs.

[0003] In the existing technology, it is possible to collect monitoring data, including task submission data, application release data, task execution data, etc., and further determine whether there are any abnormalities in these monitoring data based on the alarm conditions configured by the system, and generate alarm information when abnormalities are detected in the monitoring data to ensure the stability of the cluster.

[0004] However, the above method lacks statistical analysis of the overall operation of the system, which makes it difficult to timely discover and respond to errors or potential problems in the system, resulting in low detection accuracy and incompleteness. In addition, it judges whether the system is abnormal based on the alarm conditions of the system configuration, which has low flexibility. Summary of the Invention

[0005] The present application provides a scheduling system stability detection method, system, electronic device and storage medium, which can perform statistical analysis on the overall operation of the system, improve the accuracy and comprehensiveness of the detection, and judge whether the system has an abnormality based on the system's own stability information, thereby improving flexibility and timeliness of detection.

[0006] In a first aspect, the present application provides a method for detecting the stability of a scheduling system, the method comprising:

[0007] Obtaining task execution data and system scheduling data corresponding to the scheduling system; wherein the task execution data is used to indicate the operating status of the scheduling system when executing the task, and the system scheduling data is used to indicate completed tasks;

[0008] Calculating system stability information based on the task execution data and the system scheduling data;

[0009] If the stability information of the system does not meet the requirements, the alarm module is called to send a prompt message of the corresponding level.

[0010] Optionally, calculating system stability information based on the task execution data and system scheduling data includes:

[0011] Calculating system indicators based on the task execution data, wherein the system indicators are used to indicate the health of the system;

[0012] Calculating a production index based on the system scheduling data, wherein the production index is used to indicate the degree of completion of the task;

[0013] System stability information is calculated based on the system indicator and the production indicator.

[0014] Optionally, the task execution data includes the host's central processing unit (CPU) usage, memory usage, load value, the number of tasks being executed by the scheduling system, and the maximum number of executable tasks preset by the scheduling system; the system scheduling data includes a set of completed tasks within a first preset time period, the number of failed tasks within the first preset time period, a set of completed tasks within a second preset time period, and the start and end time of each executed task; the first preset time period is less than the second preset time period; obtaining the task execution data and system scheduling data corresponding to the scheduling system includes:

[0015] Collecting data related to task execution in the system, and splitting the collected data based on the execution results to obtain a first portion of data according to the splitting results; the first portion of data includes a central processing unit (CPU) usage rate, a memory usage rate, a load value, a number of failed tasks, and a number of successful tasks of the host at at least one moment;

[0016] Obtaining relevant data of the system scheduling tasks based on the user data packet protocol at preset intervals to obtain a second portion of data; the second portion of data includes the number of tasks being executed by the scheduling system and the maximum number of executable tasks preset by the scheduling system;

[0017] The first part of data and the second part of data are aggregated, and task execution data and system scheduling data corresponding to the scheduling system are obtained from the aggregated data.

[0018] Optionally, the system indicator includes a system pressure value and system remaining resources; the system pressure value is used to indicate the additional pressure on the system; the system remaining resources are used to indicate the remaining amount of available resources in the system; the system pressure value is determined by the following formula:

[0019] P=L / (1-C)(1-M)

[0020] The remaining system resources are determined by the following formula:

[0021] R=PM-PC

[0022] Among them, P represents the system pressure value, R represents the remaining system resources, C represents the host's central processing unit CPU usage, M represents the memory usage, L represents the load value, PC represents the number of tasks being executed by the scheduling system, and PM represents the maximum number of executable tasks preset by the scheduling system.

[0023] Optionally, the production indicators include a task failure rate and a task on-time completion rate; the task failure rate indicates the percentage of tasks that failed within a certain time period; the task on-time completion rate indicates the percentage of tasks that were completed within a certain time period; and calculating the production indicators based on the system scheduling data includes:

[0024] Calculating the ratio of the number of failed tasks within the first preset time period to the number of tasks in the task set that have been completed within the first preset time period to obtain a task failure rate;

[0025] Classifying the set of tasks completed within the second preset time period according to task type to obtain M first task sets, and classifying the set of tasks completed within the first preset time period according to task type to obtain N second task sets; where M and N are positive integers greater than 1;

[0026] The task timeliness rate is calculated based on the task completion time corresponding to each execution task in the M first task sets and the N second task sets and the number of tasks in the task sets that have been completed within the first preset time period; wherein the task completion time is the difference between the start and end time of each execution task.

[0027] Optionally, calculating the task timely rate based on the task availability time corresponding to each executed task in the M first task sets and the N second task sets and the number of tasks in the task set that have been completed within the first preset time period includes:

[0028] For each first task set, calculating the mean and standard deviation of task availability times corresponding to all executed tasks in the first task set; using the mean and standard deviation to set an interval based on a preset relationship, and determining the number of tasks in a second task set of the same type as the first task set whose task availability times fall within the interval;

[0029] The ratio of the sum of the number of tasks corresponding to the N second task sets to the number of tasks in the task set that have been completed within the first preset time period is calculated to obtain the task timeliness rate.

[0030] Optionally, the system stability information is determined by the following formula:

[0031] W=(R / P)*(T%*(1-F%)*100)

[0032] Among them, W represents system stability information, P represents system pressure value, R represents system remaining resources, T% represents task failure rate, and F% represents task timeliness rate.

[0033] Optionally, the method further includes:

[0034] Determining whether the system stability information is greater than a first preset threshold; if so, proving that the system stability is a normal indicator;

[0035] If not, it is determined whether the system stability information is greater than a second preset threshold, and a system stability index is determined according to the determination result.

[0036] Optionally, a system stability index is determined based on the judgment result, including:

[0037] If the system stability information is greater than a second preset threshold, determining that the system stability is a secondary indicator;

[0038] If the system stability information is less than a second preset threshold, the system stability is determined to be a third-level indicator.

[0039] Optionally, the method further includes:

[0040] When the system stability is determined to be a normal indicator, it is determined whether there is local information of task execution failure or task execution timeout in the system. If so, the system stability is determined to be a first-level indicator;

[0041] Correspondingly, if the stability information of the system does not meet the requirements, the alarm module is called to send prompt information of the corresponding level, including:

[0042] If the system stability is a first-level indicator, the email alarm module is called to send a first prompt message in the form of an email;

[0043] If the system stability is a secondary indicator, the email alarm module and the SMS alarm module are called to simultaneously send a second prompt message in the form of a corresponding email and SMS;

[0044] If the system stability is a third-level indicator, the email alarm module, the SMS alarm module and the phone alarm module are called to simultaneously send a third prompt message in the form of a corresponding email, SMS and phone call.

[0045] In a second aspect, the present application further provides a scheduling system stability detection system, the system comprising a statistical analysis module and an alarm module; wherein the statistical analysis module comprises a statistical module and an analysis integration module;

[0046] The statistical module is used to obtain task execution data and system scheduling data corresponding to the scheduling system; wherein the task execution data is used to indicate the operating status of the scheduling system when executing tasks, and the system scheduling data is used to indicate completed tasks;

[0047] The analysis and integration module is used to calculate system stability information based on the task execution data and system scheduling data;

[0048] The alarm module is used to call the alarm module to send prompt information of corresponding level when the stability information of the system does not meet the requirements.

[0049] In a third aspect, the present application also provides an electronic device comprising: a processor, a memory, and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor, and the computer program includes instructions for executing the scheduling system stability detection method as described in any one of the first aspects.

[0050] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the scheduling system stability detection method as described in any one of the first aspects.

[0051] In summary, the present application provides a method, system, electronic device and storage medium for detecting the stability of a scheduling system, which can obtain the task execution data and system scheduling data corresponding to the scheduling system; wherein, the task execution data is used to indicate the operating status of the scheduling system when executing the task, and the system scheduling data is used to indicate the completed task; further, the system stability information can be calculated based on the obtained task execution data and system scheduling data; further, it is judged whether the calculated stability information meets the requirements, and if it does not meet the requirements, the alarm module is called to send a prompt message of the corresponding level. In this way, the overall operation of the system can be statistically analyzed to improve the accuracy and comprehensiveness of the detection. Moreover, the present application judges whether the system is abnormal based on the stability information of the system itself, thereby improving flexibility and timeliness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the architecture of a big data job scheduling system provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a method for detecting the stability of a scheduling system provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the architecture of a scheduling system stability detection system provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the architecture of a statistical analysis module provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the architecture of an alarm module provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the structure of a scheduling system stability detection system provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0061] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0063] To facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first device and the second device are merely used to distinguish between different devices and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean that they are different.

[0064] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0065] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0066] The embodiments of the present application are described below with reference to the accompanying drawings. Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. The scheduling system stability detection method provided by the present application can be applied to the following examples: Figure 1 In the application scenario shown. The application scenario includes: user 101, scheduling system platform 102, first host 103, second host 104 and third host 105, wherein the scheduling system platform 102 can receive the task instruction sent by the terminal device of user 101. Further, the scheduling system platform 102 sends the task to each host in the system for processing based on the task instruction. Accordingly, the scheduling system platform 102 can also obtain the operating status of each host in the system and the relevant data of each host performing the task. For example, the scheduling system platform 102 can obtain the operating status of the first host 103, the second host 104 and the third host 105 and the relevant data of the above hosts performing the task. Further, the stability of the system, that is, whether an abnormality occurs, can be judged based on the above relevant data obtained.

[0067] It should be noted that the scheduling system includes a scheduling system platform 102, a first host 103, a second host 104, and a third host 105. When a task to be processed arrives, the scheduling system platform 102 will determine which host to send the task to for processing, and this application does not specifically limit the number of hosts.

[0068] It is understandable that the scheduling system platform 102 can also send corresponding prompt information based on the stability of the system to remind the user that an abnormality has occurred in the scheduling system, so as to facilitate timely discovery of system problems and maintenance.

[0069] In some embodiments, the job scheduling system is one of the core components of the big data development platform. It has the characteristics of diverse scenarios and complex design content, and can meet the complex dependencies between various jobs. For example, Figure 2 This is a schematic diagram of the architecture of a big data job scheduling system provided in an embodiment of the present application, such as Figure 2 As shown, the big data job scheduling system includes a task scheduling module, a task execution module and a database, wherein the task scheduling module is responsible for scheduling all job nodes in a workflow, including the splitting of job dependencies, the generation and distribution of jobs, and the preservation of job execution status and execution results; the task execution module is responsible for receiving jobs distributed from the scheduling module, and different jobs require corresponding types of execution modules; at the same time, during the execution of the task scheduling module and the task execution module, the execution process and status will be persisted to the database. For ease of understanding, the process mentioned in the embodiment of this application is the task executed by the task scheduling module, and the job is the task executed by the task execution module.

[0070] In the existing technology, it is possible to collect monitoring data, including task submission data, application release data, task execution data, etc., and further determine whether there are any abnormalities in these monitoring data based on the alarm conditions configured by the system, and generate alarm information when abnormalities are detected in the monitoring data to ensure the stability of the cluster.

[0071] However, the above method lacks statistical analysis of the overall operation of the system, which makes it difficult to timely discover and respond to errors or potential problems in the system, resulting in low detection accuracy and incompleteness. In addition, it judges whether the system is abnormal based on the alarm conditions of the system configuration, which has low flexibility.

[0072] Therefore, an embodiment of the present application provides a scheduling system stability detection method, which can collect data from the task scheduling module and the task execution module from the database, and further calculate the quantitative index of system stability, that is, system stability information, based on the collected data, and compare the quantitative index value with the set threshold value, and use the comparison result to determine whether to call the alarm module to issue an alarm.

[0073] The technical solution of the present application is described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0074] Figure 3 This is a flow chart of a method for detecting the stability of a scheduling system provided in an embodiment of the present application. Figure 3 As shown, the method of this embodiment may include:

[0075] S301. Obtain task execution data and system scheduling data corresponding to the scheduling system; wherein the task execution data is used to indicate the operating status of the scheduling system when executing a task, and the system scheduling data is used to indicate a completed task.

[0076] In an embodiment of the present application, the task execution data includes the host's CPU usage, memory usage, load value, the number of tasks being executed by the scheduling system, and the maximum number of executable tasks preset by the scheduling system; the system scheduling data includes the set of completed tasks within the first preset time period, the number of failed tasks within the first preset time period, the set of completed tasks within the second preset time period, and the start and end time of each executed task.

[0077] Among them, the host's central processing unit (CPU) usage rate can refer to the CPU resources occupied by the programs running on the host, indicating the status of the programs running on the host at a certain point in time. The higher the CPU usage rate, the more programs the host runs at this time, and vice versa.

[0078] Memory usage refers to the memory consumed by a process to execute tasks, that is, the memory occupied by storing calculation data in the CPU and data exchanged with external storage such as hard disk.

[0079] The load value may refer to the number of tasks queued for execution at a certain moment in the system that require CPU resources.

[0080] The number of tasks being executed by the scheduling system may refer to the number of tasks being executed by the scheduling system at a certain moment, and the maximum number of executable tasks preset by the scheduling system may refer to the maximum number of tasks that can be executed by the scheduling system at a certain moment.

[0081] The set of completed tasks within the first preset time period may refer to the set of all tasks that have been run by the system within a certain time period, including tasks that have been successfully executed and tasks that have failed to execute. For example, the set of completed tasks within the first preset time period may be the set of tasks that have been completed in the past hour.

[0082] The number of failed tasks within the first preset time period may refer to the number of failed tasks executed by the system within a certain time period. For example, the number of failed tasks within the first preset time period may be the number of failed tasks in the past hour.

[0083] The definition of the set of completed tasks within the second preset time period is similar to that of the set of completed tasks within the first preset time period and will not be repeated here. However, the first preset time period is smaller than the second preset time period. For example, the set of completed tasks within the second preset time period may be the set of tasks completed in the past week.

[0084] It should be noted that the embodiment of the present application does not limit the specific values of the first preset time period and the second preset time period. However, it must be satisfied that the first preset time period is less than the second preset time period.

[0085] The start and end time of each executed task may refer to the time recorded when each task in the system starts to be executed and the time recorded after the task is completed.

[0086] For example, in Figure 1 In the application scenario, the scheduling system platform 102 can obtain the task execution data and system scheduling data corresponding to the first host 103, the second host 104 and the third host 105 from the database. For example, the CPU usage, memory usage, load value, the number of tasks being executed by the scheduling system, the maximum number of executable tasks preset by the scheduling system, the set of tasks completed in the past hour, the number of failed tasks in the past hour, the set of tasks completed in the past week, and the start and end time of each executed task in the set can be obtained.

[0087] S302: Calculate system stability information based on the task execution data and system scheduling data.

[0088] In this step, the system stability information refers to a certain stable state exhibited by the scheduling system, which is used to reflect whether the scheduling system has any abnormalities, that is, whether system errors or problems have occurred.

[0089] For example, after collecting statistics on the data of system scheduling and task execution corresponding to the scheduling system, system stability information can be obtained through certain algorithm analysis. Furthermore, the system stability information can indicate system errors or problems.

[0090] S303: If the stability information of the system does not meet the requirements, call the alarm module to send prompt information of the corresponding level.

[0091] In an embodiment of the present application, the requirement may refer to an indicator set by the scheduling system for determining whether an error or problem occurs in the scheduling system. The level of stability of the system can be determined by setting a preset threshold and comparing it with the calculated system stability information. For example, when the system stability is a first-level indicator, it is determined that there is a problem of task execution failure or task execution timeout in the system.

[0092] The alarm module consists of three parts: email, SMS, and phone. These modules are used to send alerts to users when a system problem of a specific severity occurs. Each module is configured with the alerter's information, such as their device ID, email address, and phone number. When the system triggers an alarm module, it invokes the corresponding alarm mode to generate an alert.

[0093] It should be noted that the more serious the system problem is, the more alarm channels the alarm module will be triggered to call, and the wider the alarm range will be, so that the more serious the problem is, the more timely it can be handled.

[0094] The prompt information may refer to the information set by the system to prompt the user that the system has a problem of the corresponding level. Different levels correspond to different prompt information. For example, when the system stability index reaches the warning state, a level 2 alarm will be issued, and prompt information will be sent to the alarm persons configured in the two modules via email and text message at the same time. The prompt information is "The system stability is a level 2 indicator, the CPU memory is insufficient, and the problem is more serious"; when the system stability index reaches the level 1 alarm, only a task in the system fails to execute, then only the prompt information is sent to the alarm person configured in the email alarm module through the email alarm module, and the prompt information is "The system stability is a level 1 indicator, and task 1 fails to execute."

[0095] It should be noted that the prompt information can be displayed on the user terminal device in the form of a message prompt box, or can be sent to the user terminal device in the form of voice, and this embodiment of the present application does not specifically limit this.

[0096] For example, in Figure 1 In the application scenario, if the system stability information obtained by the scheduling system platform 102 after calculating the task execution data and system scheduling data corresponding to the first host 103, the second host 104 and the third host 105 does not meet the requirements, for example, by judging that the stability of the system is a first-level indicator, the email alarm module can be called to send a prompt message to the alarm person configured in the email alarm module, and the prompt message is "System stability is a first-level indicator, task 1 failed to execute."

[0097] Therefore, the scheduling system stability detection method provided in this application can perform statistical analysis on the overall operation of the system by obtaining system scheduling data and task execution data, thereby improving the accuracy and comprehensiveness of the detection. It can also calculate the system's own stability information based on the above information, and further use it to determine whether the system has abnormalities, thereby improving flexibility and timeliness of detection.

[0098] In order to test the stability of the dispatching system, this application Figure 2 The statistical analysis module and the alarm module are added to the architecture shown. Figure 4 A schematic diagram of the architecture of a scheduling system stability detection system provided in an embodiment of the present application; Figure 4 As shown, the scheduling system stability detection system includes a task scheduling module, a task execution module, a database, a statistical analysis module and an alarm module, wherein the statistical analysis module is used to collect data of the task scheduling module and the task execution module from the database, and obtain quantitative indicators of system stability (i.e., system stability information) by analyzing the above data. Furthermore, it is determined whether to call the alarm module to issue an alarm based on the result of comparing the indicator value with the set threshold value.

[0099] Optionally, calculating system stability information based on the task execution data and system scheduling data includes:

[0100] Calculating system indicators based on the task execution data, wherein the system indicators are used to indicate the health of the system;

[0101] Calculating a production index based on the system scheduling data, wherein the production index is used to indicate the degree of completion of the task;

[0102] System stability information is calculated based on the system indicator and the production indicator.

[0103] For example, Figure 5 A schematic diagram of the architecture of a statistical analysis module provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the statistical analysis module includes a statistical module, an analysis module and an integration module, wherein the analysis module includes a system indicator analysis module and a production indicator analysis module; specifically, the statistical module consists of two parts, the first part is responsible for collecting data from the database, and retaining the collected data after splitting; the second part embeds the acquisition module in the program code, uses the User Datagram Protocol (UDP) to send data to the data acquisition server, and retains the data in the time series database after processing. Furthermore, the two parts of data are used for the data of the system indicator analysis module and the data of the production indicator analysis module to be sent to the system indicator analysis module and the production indicator analysis module respectively. The system indicator analysis module is used to calculate the system indicators according to the task execution data, the production indicator analysis module is used to calculate the production indicators according to the system scheduling data, and the integration module is used to calculate the system stability information according to the system indicators and the production indicators.

[0104] Therefore, the system index and the production index are calculated respectively based on different data, and the system stability information is calculated through the two calculated indexes, thereby improving the calculation rate and the accuracy of the calculation.

[0105] Optionally, the task execution data includes the host's central processing unit (CPU) usage, memory usage, load value, the number of tasks being executed by the scheduling system, and the maximum number of executable tasks preset by the scheduling system; the system scheduling data includes a set of completed tasks within a first preset time period, the number of failed tasks within the first preset time period, a set of completed tasks within a second preset time period, and the start and end time of each executed task; the first preset time period is less than the second preset time period; obtaining the task execution data and system scheduling data corresponding to the scheduling system includes:

[0106] Collecting data related to task execution in the system, and splitting the collected data based on the execution results to obtain a first portion of data according to the splitting results; the first portion of data includes a central processing unit (CPU) usage rate, a memory usage rate, a load value, a number of failed tasks, and a number of successful tasks of the host at at least one moment;

[0107] Obtaining relevant data of the system scheduling tasks based on the user data packet protocol at preset intervals to obtain a second portion of data; the second portion of data includes the number of tasks being executed by the scheduling system and the maximum number of executable tasks preset by the scheduling system;

[0108] The first part of data and the second part of data are aggregated, and task execution data and system scheduling data corresponding to the scheduling system are obtained from the aggregated data.

[0109] In this step, the collected relevant data is split based on the execution results. A series of data can be split based on whether the task is executed successfully or failed. It can be divided into any data of failed task execution and data of successful task execution. It can also be split based on the status of task execution, start time, etc. The embodiment of the present application does not make specific limitations on this. The execution data of each task can include multiple indicators, and the splitting can extract the required data based on any one indicator.

[0110] In the embodiment of the present application, the preset time refers to the periodic time set by the system, which is used to periodically obtain relevant data of the system scheduling tasks, that is, the system can obtain relevant data of the system scheduling tasks every 1 minute. The embodiment of the present application does not limit the specific value of the preset time.

[0111] For example, in Figure 1In the application scenario, the scheduling system platform 102 can collect data related to task execution in the database, and split the collected relevant data based on whether the execution is successful. According to the split, the number of failed tasks and the number of successful tasks can be obtained; further, the CPU usage, memory usage, load value, etc. of the host can also be obtained. Accordingly, the scheduling system platform 102 can use the UDP protocol to send data to the data collection server every 1 minute, and use the collection server to collect the number of tasks being executed by the scheduling system and the maximum number of executable tasks preset by the scheduling system; further, the above data are summarized, and the summarized data includes data for the system indicator analysis module and data for the production indicator analysis module.

[0112] Therefore, by processing the relevant data of the tasks in the database, the task execution data and system scheduling data corresponding to the scheduling system can be obtained for calculation, thereby improving the processing rate and the accuracy of the calculation.

[0113] Optionally, the system indicator includes a system pressure value and system remaining resources; the system pressure value is used to indicate the additional pressure on the system; the system remaining resources are used to indicate the remaining amount of available resources in the system; the system pressure value is determined by the following formula:

[0114] P=L / (1-C)(1-M)

[0115] The remaining system resources are determined by the following formula:

[0116] R=PM-PC

[0117] Among them, P represents the system pressure value, R represents the remaining system resources, C represents the host's central processing unit CPU usage, M represents the memory usage, L represents the load value, PC represents the number of tasks being executed by the scheduling system, and PM represents the maximum number of executable tasks preset by the scheduling system.

[0118] For example, assuming that the system obtains C to 50%, M to 30%, L to 10, PC to 5, and PM to 20, the system indicator analysis module in the scheduling system can use P = L / (1-C)(1-M) = 10 / (1-50%)*(1-30%) = 100 to obtain a system pressure value of 100. At the same time, R = PM-PC = 20-5 = 15 can be used to obtain a system remaining resource of 15.

[0119] Therefore, the calculation speed and accuracy are greatly improved by calculating the system pressure value and the system remaining resources through the above formula.

[0120] Optionally, the production indicators include a task failure rate and a task on-time completion rate; the task failure rate indicates the percentage of tasks that failed within a certain time period; the task on-time completion rate indicates the percentage of tasks that were completed within a certain time period; and calculating the production indicators based on the system scheduling data includes:

[0121] Calculating the ratio of the number of failed tasks within the first preset time period to the number of tasks in the task set that have been completed within the first preset time period to obtain a task failure rate;

[0122] Classifying the set of tasks completed within the second preset time period according to task type to obtain M first task sets, and classifying the set of tasks completed within the first preset time period according to task type to obtain N second task sets; where M and N are positive integers greater than 1;

[0123] The task timeliness rate is calculated based on the task completion time corresponding to each execution task in the M first task sets and the N second task sets and the number of tasks in the task sets that have been completed within the first preset time period; wherein the task completion time is the difference between the start and end time of each execution task.

[0124] For example, the data set to enter the production indicator analysis module includes the set of tasks that have been completed in the past hour, recorded as TList; the number of failed tasks in the past hour, recorded as FC; and the set of tasks that have been completed in the past week, recorded as THList; among them, TList is {TList(1), TList(2), TList(3), TList(4), TList(5), TList(6)}, FC is 3, and THList is {THList(1)…THList(36)}.

[0125] Specifically, the task failure rate is: F% = FC / COUNT(TList), where COUNT is a function for finding the total number of elements in the task set, then F% = 3 / 6 = 50%. When the calculation task has a timely rate, it is necessary to classify the tasks in THList according to the task type. For example, the task type of SQL type is classified into one category, and the task type of Python type is classified into another category. Then {THList(1)…THList(36)} in THList is divided into two categories, and two first task sets are obtained, namely {THList(1)…THList(18)} and {THList(19)…THList(36)}. Further, based on the above task type, {TList (1), TList (2), TList (3), TList (4), TList (5), TList (6)}, and obtain two second task sets, namely {TList (1), TList (3), TList (4), TList (6)} and {TList (2), TList (5)}; among them, {THList (1)…THList (18)} and {TList (1), TList (3), TList (4), TList (6)} correspond to SQL type, and {THList (19)…THList (36)} and {TList (2), TList (5)} correspond to Python type.

[0126] It is understandable that THList can be divided into 3, 4, etc. categories based on task types, and its classification types can be greater than the classification types of TList. The above is only an example description, and the embodiments of this application do not make specific limitations on this.

[0127] Furthermore, the task availability rate can be calculated based on the task availability time corresponding to each execution task in the two first task sets and the two second task sets and COUNT(THList); wherein the task availability time is the difference between the start and end time of each execution task.

[0128] Therefore, the above method can be used to calculate the task failure rate and task timeliness rate based on the data entering the production index analysis module, obtain the required production indicators, and improve the calculation rate.

[0129] Optionally, calculating the task timely rate based on the task availability time corresponding to each executed task in the M first task sets and the N second task sets and the number of tasks in the task set that have been completed within the first preset time period includes:

[0130] For each first task set, calculating the mean and standard deviation of task availability times corresponding to all executed tasks in the first task set; using the mean and standard deviation to set an interval based on a preset relationship, and determining the number of tasks in a second task set of the same type as the first task set whose task availability times fall within the interval;

[0131] The ratio of the sum of the number of tasks corresponding to the N second task sets to the number of tasks in the task set that have been completed within the first preset time period is calculated to obtain the task timeliness rate.

[0132] Preferably, in the embodiment of the present application, the interval is set to [mean - 3 variance, mean + 3 variance]. This interval is an interval that has been proven by a large number of experiments and is reasonable. It can be used to screen reasonable task availability times. However, the upper and lower limits of the interval can also be set to other values. This application does not limit this. The interval can be used to screen reasonable task availability times.

[0133] For example, for each type of task in THList, the expected (i.e., mean) E(TH) and standard deviation D(TH) of the task availability time of the type are calculated respectively. Furthermore, the task availability time T is calculated for each element in TList[i], and the number of tasks TC whose task availability time T corresponds to each element in TList[i] is determined to be in the interval [E-3D, E+3D]. Furthermore, the task availability rate can be calculated using the following formula, which is T%=TC / COUNT(TList); for example, if the calculated E(TH[l]) corresponding to {THList(1)…THList(18)} is 2 and the standard deviation D(TH[l]) is 0.1, if the calculated E(TH[2 ]) is 3, the standard deviation D(TH[l]) is 0.2, and the T corresponding to TList(1) is 2, the T corresponding to TList(2) is 2.1, the T corresponding to TList(3) is 2.3, the T corresponding to TList(4) is 3.2, the T corresponding to TList(5) is 3.4, and the T corresponding to TList(6) is 2. It can be determined that the number of tasks TC[l] in the interval [E(TH[l])-3D(TH[l]), E(TH[l])+3D(TH[l])] is 2, and the number of tasks TC[2] in the interval [E(TH[2])-3D(TH[2]), E(TH[2])+3D(TH[2])] is 1, so TC=TC[1]+TC[2]=3. Further, it can be calculated that T%=3 / 6=50%.

[0134] Therefore, the timeliness rate of the task can be calculated by using the above method, improving the accuracy and speed of the calculation. The above interval is an interval verified by a large number of experiments and is reasonable.

[0135] Optionally, the system stability information is determined by the following formula:

[0136] W = (R / P) * (T% * (1 - F%) * 100)

[0137] Where, W represents the system stability information, P represents the system pressure value, R represents the system remaining resources, T% represents the task failure rate, and F% represents the timeliness rate of the task.

[0138] Exemplarily, assuming that the P calculated by the system is 100, R is 15, T% is 50%, and F% is 50%, then the integration module in the scheduling system can use W = (R / P) * (T% * (1 - F%) * 100 = (15 / 100) * (50% * (1 - 50%)) * 100 = 3.75 to obtain the system stability information of 3.75.

[0139] Therefore, the system stability information can be calculated by using the above formula, improving the speed of the system calculation, and the overall operation condition of the system can also be analyzed based on this.

[0140] Optionally, the method further includes:

[0141] Judging whether the system stability information is greater than a first preset threshold; if so, it proves that the system stability is a normal index;

[0142] If not, then judge whether the system stability information is greater than a second preset threshold, and determine the system stability index according to the judgment result.

[0143] In the embodiment of the present application, the first preset threshold may refer to a preset threshold set by the system for judging whether the state of the scheduling system is normal and healthy, and this preset threshold may be 6 times the system remaining resources. The second preset threshold may refer to a preset threshold set by the system for judging whether the scheduling system has an abnormality and the corresponding level of the abnormality, and this preset threshold may be 0.224 times the system remaining resources.

[0144] In this step, define WY as the system warning point and WW as the system danger point. When W >= WY, the system is considered healthy; when WY > W >= WW, the system enters the warning state; when W < WW, the system enters the danger state. The system warning point WY is the moment when the task scheduling system starts to have system-level errors or problems, and the system danger point WW is the moment when the system can no longer operate normally or a large number of running tasks fail or time out.

[0145] Preferably, the first preset threshold and the second preset threshold are obtained through a large number of experiments, and the proof process is as follows: when the task scheduling system is running stably, C<0.5, M<0.5, L<5, F%<0.1, T%>0.93, that is, the normally operating system W>6R; when the scheduling system load increases, the C, M, and L values will increase at the same time. When C>0.8, M>0.8, and L>10, the system operating pressure is high, or when F%>0.3 and T%<0.8, the task operation failure rate is high. At this time, W<0.224R requires manual inspection of the specific reasons; therefore, in general, 6R can be used as the empirical value of the system alarm point WY, and 0.224R can be used as the empirical value of the system danger point WW. The meanings of the above letters can be found in the description of the above embodiments and will not be repeated here.

[0146] For example, in Figure 1 In the application scenario, the scheduling system platform 102 can determine whether the calculated system stability information is greater than 6 times the system remaining resources; if so, it proves that the system stability is a normal indicator; if not, it further determines whether the system stability information is greater than 0.224 times the system remaining resources, and determines the system stability index based on the judgment result.

[0147] Therefore, the present application provides a method that can be used to divide system alarm points and system danger points under normal circumstances, thereby improving the accuracy of distinguishing the stability of the scheduling system and improving the flexibility of processing.

[0148] Optionally, a system stability index is determined based on the judgment result, including:

[0149] If the system stability information is greater than a second preset threshold, determining that the system stability is a secondary indicator;

[0150] If the system stability information is less than a second preset threshold, the system stability is determined to be a third-level indicator.

[0151] For example, in Figure 1 In the application scenario, if the scheduling system platform 102 determines that the system stability information is greater than 0.224 times the system remaining resources, the system stability is determined to be a secondary indicator, and the severity level corresponding to the secondary indicator is relatively common; if the system stability information is less than 0.224 times the system remaining resources, the system stability is determined to be a tertiary indicator, and the severity level corresponding to the tertiary indicator is relatively serious.

[0152] It should be noted that the scheduling system stability detection method provided in the embodiment of the present application is divided into three levels of severity. The different levels can correspond to different alarm channels and alarm ranges. The more alarm channels there are, the wider the alarm range. However, the embodiment of the present application can also be divided into other levels that are more than three or less than three. There is no specific limitation on this and it can be determined according to the actual situation.

[0153] Therefore, the severity level of the system problem can be determined by comparing with the set threshold, which can improve the accuracy of the judgment.

[0154] Optionally, the method further includes:

[0155] When the system stability is determined to be a normal indicator, it is determined whether there is local information of task execution failure or task execution timeout in the system. If so, the system stability is determined to be a first-level indicator;

[0156] Correspondingly, if the stability information of the system does not meet the requirements, the alarm module is called to send prompt information of the corresponding level, including:

[0157] If the system stability is a first-level indicator, the email alarm module is called to send a first prompt message in the form of an email;

[0158] If the system stability is a secondary indicator, the email alarm module and the SMS alarm module are called to simultaneously send a second prompt message in the form of a corresponding email and SMS;

[0159] If the system stability is a third-level indicator, the email alarm module, the SMS alarm module and the phone alarm module are called to simultaneously send a third prompt message in the form of a corresponding email, SMS and phone call.

[0160] In the embodiment of the present application, the first-level indicator corresponds to a lower level, the second-level indicator corresponds to a normal level, which can be regarded as entering a warning state, and the third-level indicator corresponds to a higher level, which can be regarded as entering a dangerous state. The embodiment of the present application can also be divided into more levels, which is not limited to this.

[0161] Among them, the form of email can refer to sending emails to the user terminal device, the form of text messages can refer to sending text messages to the user terminal device, and the form of phone calls can refer to making phone calls to the user terminal device. The corresponding prompt information can generate different prompt information for different levels. For example, the first prompt information can be "System stability is a first-level indicator, and task 1 failed to execute", the second prompt information can be "System stability is a second-level indicator, and the problem is more serious", and the third prompt information can be "System stability is a third-level indicator, and the problem is very serious".

[0162] For example, Figure 6A schematic diagram of the architecture of an alarm module provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the alarm module consists of three parts: an email alarm module, an SMS alarm module and a telephone alarm module. Each part includes a configuration module and an execution module. Among them, the configuration module is responsible for configuring the alarm person information, and the execution module is responsible for the execution of the alarm. Specifically, the three alarm modules work together to form three levels of alarm modes: when the system stability is determined to be a first-level indicator, the email alarm module is called to send the first prompt information in the form of email; when the system stability is a second-level indicator, the email alarm module and the SMS alarm module are called to send the second prompt information in the form of corresponding email and SMS at the same time; when the system stability is a third-level indicator, the email alarm module, the SMS alarm module and the telephone alarm module are called to send the third prompt information in the form of corresponding email, SMS and telephone at the same time.

[0163] Therefore, a combined alarm module with different alarm methods is used for different situations to ensure that the more serious problems are dealt with more promptly, improve the timeliness of early warnings, and make it easier for users to discover problems.

[0164] In the aforementioned embodiments, the scheduling system stability detection method provided in the embodiments of the present application is introduced. In order to implement the various functions of the method provided in the above embodiments of the present application, the electronic device serving as the execution subject may include a hardware structure and / or a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure and a software module. Whether a particular function is implemented in the form of a hardware structure, a software module, or a hardware structure and a software module depends on the specific application and design constraints of the technical solution.

[0165] For example, Figure 7 A schematic diagram of the structure of a scheduling system stability detection system provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the system includes a statistical analysis module 710 and an alarm module 720; wherein, the statistical analysis module 710 includes a statistical module 711 and an analysis integration module 712;

[0166] The statistical module 711 is used to obtain task execution data and system scheduling data corresponding to the scheduling system; wherein the task execution data is used to indicate the operating status of the scheduling system when executing tasks, and the system scheduling data is used to indicate completed tasks;

[0167] The analysis and integration module 712 is used to calculate system stability information based on the task execution data and system scheduling data;

[0168] The alarm module 720 is configured to call an alarm module to send a prompt message of a corresponding level when the stability information of the system does not meet the requirements.

[0169] Optionally, the analysis and integration module 712 includes a system indicator analysis module, a production indicator analysis module and an integration module;

[0170] Specifically, the system indicator analysis module is used to calculate system indicators based on the task execution data, and the system indicators are used to indicate the health of the system;

[0171] The production index analysis module is used to calculate the production index based on the system scheduling data, and the production index is used to indicate the degree of completion of the task;

[0172] The integration module is used to calculate system stability information based on the system indicators and the production indicators.

[0173] Optionally, the task execution data includes the host's central processing unit (CPU) usage, memory usage, load value, the number of tasks being executed by the scheduling system, and the maximum number of executable tasks preset by the scheduling system; the system scheduling data includes a set of completed tasks within a first preset time period, the number of failed tasks within the first preset time period, a set of completed tasks within a second preset time period, and the start and end time of each executed task; the first preset time period is less than the second preset time period; the statistical module 711 is used to:

[0174] Collecting data related to task execution in the system, and splitting the collected data based on the execution results to obtain a first portion of data according to the splitting results; the first portion of data includes a central processing unit (CPU) usage rate, a memory usage rate, a load value, a number of failed tasks, and a number of successful tasks of the host at at least one moment;

[0175] Obtaining relevant data of the system scheduling tasks based on the user data packet protocol at preset intervals to obtain a second portion of data; the second portion of data includes the number of tasks being executed by the scheduling system and the maximum number of executable tasks preset by the scheduling system;

[0176] The first part of data and the second part of data are aggregated, and task execution data and system scheduling data corresponding to the scheduling system are obtained from the aggregated data.

[0177] Optionally, the system indicator includes a system pressure value and system remaining resources; the system pressure value is used to indicate the additional pressure on the system; the system remaining resources are used to indicate the remaining amount of available resources in the system; the system pressure value is determined by the following formula:

[0178] P=L / (1-C)(1-M)

[0179] The remaining system resources are determined by the following formula:

[0180] R=PM-PC

[0181] Among them, P represents the system pressure value, R represents the remaining system resources, C represents the host's central processing unit CPU usage, M represents the memory usage, L represents the load value, PC represents the number of tasks being executed by the scheduling system, and PM represents the maximum number of executable tasks preset by the scheduling system.

[0182] Optionally, the production indicators include a task failure rate and a task on-time rate; the task failure rate is used to indicate the percentage of tasks that failed within a certain time period; the task on-time rate is used to indicate the percentage of tasks that were completed within a certain time period; the production indicator analysis module includes a first calculation unit, a classification unit, and a second calculation unit;

[0183] Specifically, the first calculation unit is used to calculate the ratio of the number of failed tasks within the first preset time period to the number of tasks in the task set that have been completed within the first preset time period to obtain a task failure rate;

[0184] The classification unit is configured to classify the set of tasks completed within the second preset time period according to task type to obtain M first task sets, and classify the set of tasks completed within the first preset time period according to task type to obtain N second task sets; wherein M and N are positive integers greater than 1;

[0185] The second calculation unit is used to calculate the task timeliness rate based on the task completion time corresponding to each execution task in the M first task sets and the N second task sets and the number of tasks in the task sets that have been completed within the first preset time period; wherein the task completion time is the difference between the start and end time of each execution task.

[0186] Optionally, the second computing unit is specifically configured to:

[0187] For each first task set, calculating the mean and standard deviation of task availability times corresponding to all executed tasks in the first task set; using the mean and standard deviation to set an interval based on a preset relationship, and determining the number of tasks in a second task set of the same type as the first task set whose task availability times fall within the interval;

[0188] The ratio of the sum of the number of tasks corresponding to the N second task sets to the number of tasks in the task set that have been completed within the first preset time period is calculated to obtain the task timeliness rate.

[0189] Optionally, the system stability information is determined by the following formula:

[0190] W=(R / P)*(T%*(1-F%)*100)

[0191] Among them, W represents system stability information, P represents system pressure value, R represents system remaining resources, T% represents task failure rate, and F% represents task timeliness rate.

[0192] Optionally, the system further includes a judgment module, wherein the judgment module is configured to:

[0193] Determining whether the system stability information is greater than a first preset threshold; if so, proving that the system stability is a normal indicator;

[0194] If not, it is determined whether the system stability information is greater than a second preset threshold, and a system stability index is determined according to the determination result.

[0195] Optionally, the judgment module is specifically configured to:

[0196] If the system stability information is greater than a second preset threshold, determining that the system stability is a secondary indicator;

[0197] If the system stability information is less than a second preset threshold, the system stability is determined to be a third-level indicator.

[0198] Optionally, the judgment module is further configured to:

[0199] When the system stability is determined to be a normal indicator, it is determined whether there is local information of task execution failure or task execution timeout in the system. If so, the system stability is determined to be a first-level indicator;

[0200] Accordingly, the alarm module 720 is used to:

[0201] If the system stability is a first-level indicator, the email alarm module is called to send a first prompt message in the form of an email;

[0202] If the system stability is a secondary indicator, the email alarm module and the SMS alarm module are called to simultaneously send a second prompt message in the form of a corresponding email and SMS;

[0203] If the system stability is a third-level indicator, the email alarm module, the SMS alarm module and the phone alarm module are called to simultaneously send a third prompt message in the form of a corresponding email, SMS and phone call.

[0204] The specific implementation principles and effects of the scheduling system stability detection system provided in the embodiments of the present application can be found in the relevant descriptions and effects corresponding to the above embodiments, and will not be elaborated here.

[0205] The embodiment of the present application also provides a structural diagram of an electronic device, Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 802 and a memory 801 communicatively connected to the processor; the memory 801 stores a computer program; the processor 802 executes the computer program stored in the memory 801, so that the processor 802 executes the method described in any of the above embodiments.

[0206] The memory 801 and the processor 802 may be connected via a bus 803 .

[0207] An embodiment of the present application further provides a computer-readable storage medium, which stores computer program execution instructions. When the computer program execution instructions are executed by a processor, they are used to implement the method described in any of the aforementioned embodiments of the present application.

[0208] An embodiment of the present application further provides a chip for executing instructions, which is used to execute the method in any of the aforementioned embodiments as executed by an electronic device in any of the aforementioned embodiments of the present application.

[0209] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the method described in any of the aforementioned embodiments of the present application executed by an electronic device.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0211] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0212] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The above-mentioned modules may be implemented in the form of hardware or hardware plus software functional units.

[0213] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0214] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules in the processor.

[0215] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0216] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0217] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0218] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0219] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for detecting the stability of a dispatching system, characterized in that: The method comprises: Obtain task execution data and system scheduling data corresponding to the scheduling system; wherein the task execution data is used to indicate the operating status of the scheduling system when executing tasks, and the system scheduling data is used to indicate completed tasks; the task execution data includes the host's central processing unit CPU usage, memory usage, load value, the number of tasks being executed by the scheduling system, and the maximum number of executable tasks preset by the scheduling system; the system scheduling data includes a set of completed tasks within a first preset time period, the number of failed tasks within the first preset time period, a set of completed tasks within a second preset time period, and the start and end times of each executed task; the first preset time period is less than the second preset time period; Calculating system stability information based on the task execution data and the system scheduling data; If the stability information of the system does not meet the requirements, the alarm module is called to send a prompt message of the corresponding level; the system stability information is calculated based on the task execution data and the system scheduling data, including: Calculating system indicators based on the task execution data, wherein the system indicators are used to indicate the health of the system; Calculating a production index based on the system scheduling data, wherein the production index is used to indicate the degree of completion of the task; Calculating system stability information based on the system indicator and the production indicator; The obtaining of task execution data and system scheduling data corresponding to the scheduling system includes: Collecting data related to task execution in the system, and splitting the collected data based on the execution results to obtain a first portion of data according to the splitting results; the first portion of data includes a central processing unit (CPU) usage rate, a memory usage rate, a load value, a number of failed tasks, and a number of successful tasks of the host at at least one moment; Obtaining relevant data of the system scheduling tasks based on the user data packet protocol at preset intervals to obtain a second portion of data; the second portion of data includes the number of tasks being executed by the scheduling system and the maximum number of executable tasks preset by the scheduling system; Summarizing the first part of data and the second part of data, and obtaining task execution data and system scheduling data corresponding to the scheduling system from the summarized data; The system indicators include a system pressure value and a system remaining resource; the system pressure value is used to indicate the additional pressure on the system; the system remaining resource is used to indicate the remaining amount of available system resources; the system pressure value is determined by the following formula: P=L / (1-C)(1-M) The remaining system resources are determined by the following formula: R=PM-PC Among them, P represents the system pressure value, R represents the remaining system resources, C represents the host's central processing unit CPU usage, M represents the memory usage, L represents the load value, PC represents the number of tasks being executed by the scheduling system, and PM represents the maximum number of executable tasks preset by the scheduling system.

2. The method according to claim 1, characterized in that The production indicators include task failure rate and task timely rate; the task failure rate is used to indicate the percentage of tasks that failed in a certain time period; the task timely rate is used to indicate the percentage of tasks that were completed in a certain time period; Calculate production indicators based on the system scheduling data, including: Calculating the ratio of the number of failed tasks within the first preset time period to the number of tasks in the task set that have been completed within the first preset time period to obtain a task failure rate; Classifying the set of tasks completed within the second preset time period according to task type to obtain M first task sets, and classifying the set of tasks completed within the first preset time period according to task type to obtain N second task sets; where M and N are positive integers greater than 1; The task timeliness rate is calculated based on the task completion time corresponding to each execution task in the M first task sets and the N second task sets and the number of tasks in the task sets that have been completed within the first preset time period; wherein the task completion time is the difference between the start and end time of each execution task.

3. The method according to claim 2, characterized in that Calculating the task availability rate based on the task availability time corresponding to each execution task in the M first task sets and the N second task sets and the number of tasks in the task set that have been completed within the first preset time period includes: For each first task set, calculating the mean and standard deviation of task availability times corresponding to all executed tasks in the first task set; using the mean and standard deviation to set an interval based on a preset relationship, and determining the number of tasks in a second task set of the same type as the first task set whose task availability times fall within the interval; The ratio of the sum of the number of tasks corresponding to the N second task sets to the number of tasks in the task set that have been completed within the first preset time period is calculated to obtain the task timeliness rate.

4. The method according to claim 3, characterized in that The system stability information is determined by the following formula: W=(R / P)*(T%*(1-F%)*100) Among them, W represents system stability information, P represents system pressure value, R represents system remaining resources, T% represents task failure rate, and F% represents task timeliness rate.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Determining whether the system stability information is greater than a first preset threshold; If so, it proves that the system stability is a normal indicator; If not, it is determined whether the system stability information is greater than a second preset threshold, and a system stability index is determined according to the determination result.

6. The method according to claim 5, characterized in that Determine the system stability index based on the judgment results, including: If the system stability information is greater than a second preset threshold, determining that the system stability is a secondary indicator; If the system stability information is less than a second preset threshold, the system stability is determined to be a third-level indicator.

7. The method according to claim 6, characterized in that Also includes: When the system stability is determined to be a normal indicator, it is determined whether there is local information of task execution failure or task execution timeout in the system. If so, the system stability is determined to be a first-level indicator; Correspondingly, if the stability information of the system does not meet the requirements, the alarm module is called to send prompt information of the corresponding level, including: If the system stability is a first-level indicator, the email alarm module is called to send a first prompt message in the form of an email; If the system stability is a secondary indicator, the email alarm module and the SMS alarm module are called to simultaneously send a second prompt message in the form of a corresponding email and SMS; If the system stability is a third-level indicator, the email alarm module, the SMS alarm module and the phone alarm module are called to simultaneously send a third prompt message in the form of a corresponding email, SMS and phone call.

8. A dispatching system stability detection system, characterized in that: The system includes a statistical analysis module and an alarm module; wherein the statistical analysis module includes a statistical module and an analysis integration module; The statistical module is used to obtain task execution data and system scheduling data corresponding to the scheduling system; wherein, the task execution data is used to indicate the operating status of the scheduling system when executing tasks, and the system scheduling data is used to indicate completed tasks; the task execution data includes the host's central processing unit CPU usage, memory usage, load value, the number of tasks being executed by the scheduling system, and the maximum number of executable tasks preset by the scheduling system; the system scheduling data includes a set of completed tasks within a first preset time period, the number of failed tasks within the first preset time period, a set of completed tasks within a second preset time period, and the start and end time of each executed task; the first preset time period is less than the second preset time period; The analysis and integration module is used to calculate system stability information based on the task execution data and system scheduling data; The alarm module is used to call the alarm module to send prompt information of corresponding level when the stability information of the system does not meet the requirements; The analysis and integration module is further configured to calculate system indicators based on the task execution data, wherein the system indicators are used to indicate the health of the system; calculate production indicators based on the system scheduling data, wherein the production indicators are used to indicate the degree of completion of the task; and calculate system stability information based on the system indicators and the production indicators; The statistical module is further configured to collect data related to task execution in the system, and to split the collected data based on the execution results, thereby obtaining a first portion of data according to the split results; the first portion of data includes the CPU usage rate, memory usage rate, load value, number of failed tasks, and number of successful tasks of the host computer at at least one moment; obtain data related to system scheduling tasks based on the user data packet protocol at preset intervals to obtain a second portion of data; the second portion of data includes the number of tasks being executed by the scheduling system and the maximum number of executable tasks preset by the scheduling system; the first portion of data and the second portion of data are aggregated, and task execution data and system scheduling data corresponding to the scheduling system are obtained from the aggregated data; The system indicators include a system pressure value and system remaining resources; the system pressure value is used to indicate the additional pressure on the system; the system remaining resources are used to indicate the remaining amount of available resources in the system; the system pressure value can be determined by the following formula: P=L / (1-C)(1-M) The remaining system resources are determined by the following formula: R=PM-PC P represents the system pressure value, R represents the remaining system resources, C represents the CPU usage of the host, M represents the memory usage, L represents the load value, PC represents the number of tasks being executed by the scheduling system, and PM represents the maximum number of executable tasks preset by the scheduling system.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor, and the computer program includes instructions for executing the scheduling system stability detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the scheduling system stability detection method according to any one of claims 1 to 7.

Images