Batch monitoring method, device and equipment

Abstract

The embodiments of the present application provide a batch monitoring method, device and equipment. The method includes: obtaining the jobs planned to be run by the application according to the configuration information of the job, and generating a job running schedule; obtaining the key running indicators of the job through the running history data of the job, and generating a job running history indicator table; obtaining the jobs that have been run to calculate and display the progress of the application through the job running schedule; combining with the job running history indicator table, displaying abnormal situations and drilling down the information of abnormal jobs, and obtaining the abnormal job list of the application. The technical solution of the present application has the ability of global monitoring. By organizing the overall batch monitoring by the dimension of the scene, the overall processing progress can be controlled in real time, and the operation and maintenance efficiency can be improved; through the abnormal job list of each application in the scene, the operation and maintenance can be carried out at a fixed point; through the job running schedule, the scope of the job to be monitored is locked in advance, without waiting for the job flow to be instantiated, thereby improving the accuracy of monitoring.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a batch monitoring method, device, equipment and processor. Background Art

[0002] Traditional batch processing scheduling monitoring faces the following challenges: With the increase in business, the daily running volume is also increasing, and the relationship between jobs is becoming more and more complicated, making batch monitoring inefficient; from the perspective of global monitoring, different processing links have different importance, and it is not possible to monitor important links in a targeted manner; real-time monitoring is difficult. Due to the complexity of batch processing business, it is impossible to display the processing progress of each application in real time, obtain the abnormal conditions of the current batch processing, and display the "critical path" that affects the final target application.

[0003] To this end, the job scheduling system has the following problems in applications based on traditional batch scheduling monitoring: it basically provides fine-grained monitoring such as job flows or jobs, without displaying them from a higher dimension; some products can display the dependencies and processing conditions between all jobs on a canvas, but the disadvantage is that the user experience is poor, and it is often disorganized and difficult to reflect the actual business meaning. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a batch monitoring method, device, equipment, storage medium and processor.

[0005] To this end, a batch processing monitoring method is established that displays the processing progress and abnormal job list of each application's specific business date / batch in the scenario in real time, and supports drilling down to obtain more fine-grained job instances, which can solve at least one of the following technical problems:

[0006] (1) Traditional scheduling software mainly monitors batch processing from the perspective of job flow, which is not abstract enough. As a result, many jobs are often displayed on a single canvas, similar to a spider web. In addition, when there are many jobs, the software may not be able to refresh in real time due to the large amount of computation.

[0007] (2) Traditional scheduling software generally only provides monitoring on instantiated job flows / job instances, and does not make good use of historical operation information, such as the average operation time of the job, the normal start time of the job, etc., nor does it combine the planned data for monitoring. Users can only be informed when the job has failed, and it is impossible to more effectively combine these historical data for real-time monitoring;

[0008] (3) Traditional scheduling software generally only provides first-level monitoring and lacks drill-down capabilities, which cannot better meet operation and maintenance needs.

[0009] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a batch processing monitoring method, including: obtaining the planned running jobs of the application according to the configuration information of the job, and generating a job running schedule; obtaining the key running indicators of the job through the running history data of the job, and generating a job running history indicator table; through the job running schedule, obtaining the jobs that have been run to calculate and display the progress of the application completion; in combination with the job running history indicator table, displaying abnormal situations and drilling down the information of abnormal jobs, and obtaining a list of abnormal jobs of the application.

[0010] In an embodiment of the present application, the planned running jobs of the application are obtained according to the configuration information of the job, and a job running schedule is generated, including: identifying the end node applications of the scene, where one scene corresponds to one end node application; determining the dependency relationship between the end node application and the job of the corresponding scene; and calculating each node application step by step according to the dependency relationship between the jobs to obtain the AOE network of the scene, thereby generating an actual job list.

[0011] Furthermore, identifying the end node application of the scene includes: classifying each scene according to the configuration information of the job; and identifying the target application associated with each scene as the end node application.

[0012] Furthermore, obtaining the key operation indicator of the job through the operation history data of the job includes taking the median of the operation history data as the key operation indicator of the job.

[0013] Furthermore, the key operation indicators of the job include: the execution time of the job, the start time node of the job, and the end time node of the job.

[0014] Furthermore, the method further comprises: regularly refreshing the abnormal job list, keeping the refreshed data in a cache, and directly reading the refreshed data.

[0015] Furthermore, through the job running schedule, the jobs that have been completed are obtained to calculate and display the progress of each application, including: polling the current completion status of the jobs based on the number of jobs contained in each application in the scenario, where the real-time progress of each application = the number of jobs that have been successfully executed / the total number of jobs contained in the application in the scenario; the progress of the terminal node application is used to characterize the overall progress of the scenario.

[0016] Furthermore, the abnormal situation is displayed and the information of the abnormal job is drilled down to obtain the abnormal job list of each application, including: obtaining the path with the maximum path length from the AOE network of the scene, marking it as the critical path; according to the critical path, marking the abnormal situation, including marking job failure, job timeout and job delayed start.

[0017] Through the method of the present application, the global batch processing status is monitored in the dimension of the scene; by identifying the end application and the end application job set of the scene, the entire AOE graph is automatically calculated and constructed, thereby greatly reducing the user's configuration amount.

[0018] The second aspect of the present application provides a batch processing monitoring device, including: a first module, used to obtain the planned running jobs of the application according to the configuration information of the job, and generate a job running schedule; a second module, used to obtain the key running indicators of the job through the running history data of the job, and generate a job running history indicator table; a third module, used to obtain the jobs that have been completed through the job running schedule to calculate and display the progress of the application completion; a fourth module, used to display abnormal situations and drill down the information of abnormal jobs in combination with the job running history indicator table, and obtain a list of abnormal jobs of the application.

[0019] In an embodiment of the present application, the first module is configured to: classify each scene according to the configuration information of the job; identify the target application associated with each scene as an end node application; identify the end node application of the scene, where one scene corresponds to one end node application; determine the dependency relationship between the end node application and the job of the corresponding scene; and calculate each node application step by step according to the dependency relationship between the jobs to obtain the AOE network of the scene, thereby generating the actual job list.

[0020] Furthermore, the second module is configured to: poll the current completion status of the jobs according to the number of jobs contained in each application in the scenario, wherein the real-time progress of each application = the number of jobs that have been successfully executed / the total number of jobs contained in the application in the scenario; and use the progress of the terminal node application to represent the overall progress of the scenario.

[0021] Furthermore, the fourth module is configured to: obtain abnormal situation classification; calculate the proportion of abnormal operations through the classification of the abnormal situation; and differentiate and display according to the proportion of abnormal operations.

[0022] The technical effect of the batch processing monitoring device described in this application is the same as the batch processing monitoring method.

[0023] Another aspect of the present application provides a processor configured to execute the above batch processing monitoring method.

[0024] Another aspect of the present application provides a batch monitoring device, the device comprising a processor and a memory, wherein the processor is configured to execute the batch monitoring method.

[0025] A fifth aspect of the present application provides a computer program product, comprising a computer program, which implements the batch monitoring method when executed by a processor.

[0026] A machine-readable storage medium stores instructions, which, when executed by a processor, cause the processor to be configured to execute the batch monitoring method.

[0027] Through the above technical solution, the overall batch processing monitoring is organized through the dimension of the scene, and the overall processing progress is controlled in real time; the operation and maintenance efficiency is improved. The operation and maintenance personnel can perform operation and maintenance on these jobs at a fixed point through the list of abnormal jobs of each application in the scene, which also reduces the manpower input of the operation and maintenance personnel.

[0028] Through the technical solution of the present application, the global batch processing status is monitored in the dimension of scenes, and each scene represents a batch processing logic with business significance; by identifying the end application and the end application job set of the scene, the entire AOE graph is automatically calculated and constructed, thereby greatly reducing the user's configuration amount.

[0029] AOE is used to characterize the scene, so as to conveniently display the relationship and calculation key path of each application in the scene, and combine historical data to determine whether the current operation of the job is normal. In addition, the job operation schedule is combined to infer the set of jobs that need to be run on the specific business date / batch of the application, and the monitoring scope is locked without waiting for the job flow to complete instantiation.

[0030] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0032] Figure 1 The following schematically shows a block diagram of the prior art Control-M system architecture;

[0033] Figure 2 The flowchart of the batch processing monitoring method according to the embodiment of the present application is schematically shown;

[0034] Figure 3A A schematic diagram of a process flow for generating a job running schedule according to an embodiment of the present application is shown;

[0035] Figure 3B The following schematically shows a process diagram of generating a job operation history indicator table according to an embodiment of the present application;

[0036] Figure 3C The following schematically shows a flow chart of abnormal job list generation according to an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a scene configuration process according to an embodiment of the present application is shown schematically;

[0038] Figure 5 A schematic diagram of a scene AOE example according to an embodiment of the present application is shown;

[0039] Figure 6 The structure block diagram of the batch processing monitoring device according to the embodiment of the present application is schematically shown;

[0040] Figure 7 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0042] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0043] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0044] Figure 1 The following schematically shows a block diagram of the prior art Control-M system architecture.

[0045] See also Figure 1 The technical solution of the Control-M system realizes the separation of the three functions of management, scheduling and job execution through a three-layer architecture. Among them, the first layer of Control-M / EM is the management module, responsible for the definition / upload of jobs, operation monitoring, etc.; the second layer of Control-M / Server is the core of the entire scheduling, responsible for the instantiation of job flows, job scheduling, resource allocation, job operation management, etc.; the third layer of Control-M / Agent is the job operation node, responsible for running jobs issued by the Server, saving the job operation status, and synchronizing the job operation status with Control-M / Server. Among them, the monitoring function is mainly completed by Control-M / EM. The principle is that Control-M / EM obtains the dependency relationship between jobs from the data in Control-M / EM DB and obtains the operation status of job instances in real time from the data in Control-M / Server DB through Control-M / Server. Users can perform real-time monitoring by specifying the target job flow / job collection to be monitored.

[0046] The disadvantages of this existing technology are as follows: the user needs to select all the job flows / jobs to be monitored, which requires the user to be very familiar with the full-link processing process to be monitored; the relationship between the selected jobs is displayed in a canvas with a relatively fine dimension, which cannot enable better monitoring from a global perspective; and the overall processing progress of the target job monitored by the user cannot be reflected.

[0047] For the Airflow-based scheduling system in the prior art ( Figure 1 (not shown), this technical solution mainly monitors from the perspective of the job flow (that is, DAG), and provides tree structure, graphic structure, Gantt chart and other methods for visual display, which can better monitor the job flow. The tree and graphic structures can conveniently display the running status of each job instance under the job flow, and the Gantt chart mainly analyzes the bottleneck point of the job flow by analyzing the running start and end time of each job constituting the job flow. Its essence is still monitoring from the perspective of the job flow.

[0048] The disadvantages of the prior art are as follows: the Airflow-based scheduling system provides monitoring of the job flow dimension, lacks higher-dimensional monitoring, that is, the scenario-dimensional monitoring of the present application introduced below, which is difficult to meet the requirements of global batch processing monitoring; the monitoring display only reflects the running status of the instantiated jobs, and cannot display the job flows that have not yet been instantiated.

[0049] For the scheduling system based on DolphinScheduler in the prior art ( Figure 1 (not shown), this technical solution is similar to the implementation of the scheduling system based on Airflow, and it also monitors from the perspective of workflow (same as job flow), providing a tree diagram to show the type and status of task nodes; in addition, DolphinScheduler also provides a Gantt chart to complete the analysis of workflow bottlenecks. Its technical shortcomings are the same as above.

[0050] The batch monitoring method provided in the present application can be applied, for example, in an Internet application environment. The Internet application environment includes: a terminal, a server, and a network. The terminal communicates with the server through the network. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0051] Figure 2 The flowchart of the batch monitoring method according to the embodiment of the present application is schematically shown. Figure 2 As shown, in one embodiment of the present application, a batch monitoring method is provided. This embodiment is mainly illustrated by applying the method to the above-mentioned terminal (or server), and includes the following steps:

[0052] S202, obtaining the job to be run by the application according to the configuration information of the job, and generating a job running schedule.

[0053] According to the embodiment, the application is a plurality of applications, each of which has a plurality of jobs. The application can be simply understood as a project, and the application can include a plurality of job streams, each of which has a plurality of jobs. Each job belongs to a certain application.

[0054] First, the dependencies between jobs are found based on the input and output conditions of the job initially configured by the user. When the input conditions of the job are met, the job will be started.

[0055] Then, based on the dependencies between jobs, they are converted into dependencies between applications.

[0056] S204, obtaining key operation indicators of the job through the operation history data of the job, and generating a job operation history indicator table. Specifically, the key operation indicators of the job include the execution time of the job, the start time node of the job, and the end time node of the job.

[0057] S206, obtaining the completed jobs through the job running schedule to calculate and display the progress of application completion.

[0058] S208, combining the job running history indicator table, displaying abnormal situations and drilling down on abnormal job information to obtain an abnormal job list of the application.

[0059] This application organizes overall (global) batch monitoring through the dimension of the scene, and controls the overall processing progress in real time. The operation and maintenance personnel can operate and maintain these jobs at a fixed point through the abnormal operation list of each application in the scene, which also reduces the manpower input of the operation and maintenance personnel.

[0060] Figure 3A The following schematically shows a flow chart of generating a job running schedule according to an embodiment of the present application.

[0061] See also Figure 3A , combined with Figure 2 The method flow chart of the present application is described in detail.

[0062] According to the embodiment, the method of the technical solution of the present application includes three processes as a whole: a job operation schedule generation process, a job operation historical indicator refresh process, and a scene AOE operation information refresh process (i.e., the main process). The main function of the job operation schedule is to obtain a list of jobs that need to be run every day for each application based on the configuration information (frequency) of the job. The job configuration information used here is mainly the execution frequency of the job, such as daily, weekly, monthly execution, execution at the beginning of the month, execution at the end of the month, etc., or a specific time point, such as execution at 09:00 every day.

[0063] In S301, the database scheduling configuration information is obtained. After the scenario configuration takes effect, the real-time operation information of the scenario needs to be calculated every day. In the specific implementation, the job set that needs to be run on the end node (end point) for this business date or batch is first obtained, and then the actual job list (job set) that needs to be run on this business date or batch of the scenario can be obtained by calculating forward step by step according to the dependency relationship between the jobs.

[0064] According to the embodiment, the job set here is found according to the dependency relationship of the terminal job, and belongs to all jobs under the application. It can be understood that for the terminal job to be successfully executed, it is necessary to pay attention to the upstream jobs it depends on. The job set here is the job that needs to be paid attention to, which can also be called a key job.

[0065] According to the embodiment, the current business date can be understood as the specific date to which a batch of jobs belongs. Generally, the current business date is equal to the job execution date, but there are also inconsistencies.

[0066] In this embodiment, the job running schedule is refreshed regularly in S303. Preferably, the job running schedule is refreshed once a day, or the job running schedule can be refreshed regularly according to actual conditions.

[0067] According to the embodiment, the job running schedule is set to be automatically loaded every day. Operation authority, select scheduling date, job name, job type, command execution user, job scheduling date allowed, such as daily scheduling, abnormal logic processing for jobs, and scene information configuration.

[0068] The scope of operations that need to be monitored can be locked in advance through the schedule, without waiting for the instantiation of the operation flow, which greatly improves the accuracy of monitoring.

[0069] Figure 3B The following schematically shows a flow chart of generating a job operation history indicator table according to an embodiment of the present application.

[0070] See also Figure 3B According to the embodiment, the job operation history indicator table is used to obtain the key operation indicators of the job through the historical operation data of the job, which can also be understood as important operation indicators.

[0071] According to the embodiment, at S311, the flow information of the successfully executed job is obtained, such as the execution time of the job, the start time (time node) of the job, and the end time (time node) of the job.

[0072] In S313, the job running time, start time node, and end time node are refreshed regularly according to the median method. Specifically, the calculation method uses the median of historical data. The median here refers to the median of the historical running time of the job. Some jobs may be run in batches every day, and there will be a running time every day. There will be a stored procedure (Oracle) that counts the median running time of all jobs every day.

[0073] At S315 , the job execution history indicator table is refreshed.

[0074] Figure 3C The following schematically shows a flow chart of generating a list of abnormal operations according to an embodiment of the present application.

[0075] See also Figure 3C According to the embodiment, in S321, the scene AOE refresh main process can be refreshed every 5 minutes. When it is determined that the refresh is not completed, enter S323 to obtain the currently configured scene information. In S235, obtain the set of jobs that need to be run for all applications of this business date or batch according to the job operation schedule.

[0076] According to an embodiment, at S237, the number of jobs successfully run by each application is updated, the progress information is refreshed, and the jobs that have been run to completion are obtained to calculate the progress of the application.

[0077] In S239, by judging the job running flow information and combining the job running history indicator table, the abnormal job list of each application is obtained, and further drilling of abnormal job information is supported. All data refreshed by the scene AOE refresh main process is kept in the Redis cache, and the Redis data can be directly read on the page, which also avoids the situation where the page cannot be refreshed due to excessive calculation.

[0078] Figure 4 A schematic diagram of a scene configuration process according to an embodiment of the present application is schematically shown.

[0079] See also Figure 4 A scenario represents a batch processing business logic, such as regulatory reporting, data warehouse processing, etc. The scenario itself has business meaning. The scenario configuration process is as follows:

[0080] In S401, the scenarios are classified according to the configuration information of the job. Specifically, the scenarios are determined according to the batch processing business logic of interest.

[0081] In S403, the target application associated with each scenario is identified as an end node application. Specifically, the target application associated with the scenario is identified as the terminal application, which can also be understood as an end node application. The end node application of each scenario is identified, and one scene corresponds to one end node application.

[0082] In S405, the dependency relationship between the terminal node application and the job of the corresponding scenario is determined.

[0083] According to the embodiment, the job set associated with the scenario, the business date / batch rules of interest (the rules will infer the dates and batches to be monitored based on the current natural date, such as T(1) represents the day after the current system date, T(-1) represents the day before the current system date, and the batch is composed of 4 digits, covering 1-9999, such as B(1) represents the first batch) and the calculation level are determined. The calculation level is mainly used to determine the maximum level of automatic calculation.

[0084] In S407, the node applications are calculated step by step according to the dependency relationship between the jobs to obtain the AOE network of the scene, thereby generating the actual job list.

[0085] Since the scene itself is meaningful, a scene also represents an important batch processing logic. By combing through the scene process, we can also better understand the batch processing business from a business perspective, making monitoring more targeted. During the scene configuration process, after identifying the scene, the user only needs to determine the end application, and the system will automatically calculate and build the AOE diagram of the scene, reducing the configuration complexity and being more user-friendly.

[0086] According to the embodiment, the upstream application is automatically calculated step by step based on the dependency relationship between jobs in the system and the application to which the job belongs, thereby calculating the AOE graph of the scene, which represents the processing link of the scene. After the scene configuration is completed, the actual job list can be obtained.

[0087] Figure 5 An example diagram of a scene AOE according to an embodiment of the present application is schematically shown.

[0088] See also Figure 5 , a typical scene AOE graphics is as follows Figure 5 The data that can be displayed in this application include the dependency relationship of the scene, real-time processing progress, and abnormal situation display (i.e., whether the current progress is at a normal level). The following describes the implementation methods respectively.

[0089] Through the actual job list, the jobs that have been run and completed are obtained to calculate and display the progress of each application, including: polling the current completion status of the jobs according to the number of jobs contained in each application in the scene, where the real-time progress of each application = the number of jobs that have been successfully executed / the total number of jobs contained in the application in the scene; the progress of the end node application is used to characterize the overall progress of the scene.

[0090] According to an embodiment, Figure 5 Dependency: By converting the dependencies between jobs into dependencies between applications, the application to which the upstream job belongs will have an edge pointing to the downstream application. For example, the starting application A1 is the upstream application of application A2 and application A3. For the edge of application A1, the corresponding job sets of upstream and downstream are determined.

[0091] According to an embodiment, the real-time processing progress is based on Figure 5 The job collection contained in each application in the scenario is polled regularly through the background thread to check the current completion status of these jobs. Specifically, the progress of the starting application A1 is 100%, and the progress of application A2 is 30%. The calculation method is: real-time progress of the application = number of successfully executed jobs / total number of jobs contained in the application in this scenario.

[0092] According to the embodiment, the overall progress of the scenario is represented by the progress of the end node application (end point application), for example, the progress of application A6 is 30%.

[0093] According to the embodiment, the abnormal situation includes three situations: job failure, job timeout, and delayed job start. Displaying the abnormal situation and drilling down the job instance information to obtain the abnormal job list of each application includes: obtaining the path with the maximum path length from the AOE network of the scene, marking it as a critical path; marking the abnormal situation according to the critical path, including marking job failure, job timeout, and delayed job start.

[0094] According to the embodiment, the failure of a job can be obtained in real time based on the running flow of the job; the timeout of a job and the delayed start of a job need to be obtained in combination with the historical running data. The background calculates the historical running indicators (execution duration, start time node) of each job through a fixed thread, and then compares the running status of the job in real time to see whether there are timeouts and delayed starts. In summary, the proportion of abnormal jobs = (the number of failed jobs + the number of timed jobs + the number of delayed start jobs) / the total number of jobs included in the application in this scenario.

[0095] According to an embodiment, the severity of the abnormal situation can be displayed by different colors. For example, 80-100% of the operation abnormalities correspond to dark red, 50-80% correspond to orange, 20-50% correspond to light yellow, 10-20% correspond to blue, and 0%-10% correspond to green.

[0096] According to an embodiment, when drilling down on an application in a scenario, the status of each job in the job set under the application is displayed. Referring to Table 1, the job name can be further jumped to the job instance page. In this way, the detailed processing status of each job under the current application can be displayed in more detail.

[0097] Application Name Job Stream Name Job Name Business Date batch Abnormal identification Application 1 Flowa Job1 20210101 0001 Execution failed Application 1 Flowb Job2 20210101 0001 Delayed start Application 1 Flowc Job3 20210101 0001 normal

[0098] Table 1 Scenario application drill-down information

[0099] According to an embodiment, in the above AOE graph, the path from the starting point to the end point with the maximum path length (the sum of the duration of each activity on the path) is the critical path. Figure 5In the case where A1 to A2 passes through A4 to reach the end application A6 as the critical path, the key to calculating the critical path is how to define the weight of the edge. Two strategies are provided in this application to define the weight of the edge. Strategy 1: The value with the longest running time in the job set of the upstream application associated with the edge represents the weight of the edge. For example, the upstream applications of application A4 are A2 and A3. Then, through strategy 1, it is judged that application A2 with a progress of 30% has the longest running time weight, and the critical path is obtained from A1 to A2 passing through A4 to reach the end application A6, where the running time of each job can be represented by the median running time of the past 30 dates.

[0100] According to the embodiment, strategy 2: the latest completed job time in the job set of the upstream application associated with the edge is represented (the end time of the job is also the median end time of the job in the past 30 dates). For the convenience of calculation, the latest end time needs to be converted into an integer value based on a certain time reference value to represent the weight of the edge. After determining the weight of each edge, the critical path can be calculated, and the path with the largest sum of edge weights from the starting application to the end application is determined as the critical path.

[0101] This application organizes the overall batch processing monitoring through the dimension of the scene, and controls the overall processing progress in real time; it improves the operation and maintenance efficiency. The operation and maintenance personnel can perform operation and maintenance on these jobs at a fixed point through the abnormal operation list of each application in the scene, which also reduces the manpower input of the operation and maintenance personnel.

[0102] Through the technical solution of the present application, the global batch processing status is monitored in the dimension of scenes, and each scene represents a batch processing logic with business significance; by identifying the end application and the end application job set of the scene, the entire AOE graph is automatically calculated and constructed, thereby greatly reducing the user's configuration amount.

[0103] AOE is used to characterize the scene, so as to conveniently display the relationship and calculation key path of each application in the scene, and combine historical data to determine whether the current operation of the job is normal. In addition, the job operation schedule is combined to infer the set of jobs that need to be run on the specific business date / batch of the application, and the monitoring scope is locked without waiting for the job flow to complete instantiation.

[0104] Figure 6 The structural block diagram of a batch processing monitoring device according to an embodiment of the present application is schematically shown.

[0105] In one embodiment, Figure 6 As shown, a batch processing monitoring device 600 is provided, comprising a first module 601, a second module 603, a third module 605 and a fourth module 607, wherein:

[0106] The first module 601 is used to obtain the job to be run by the application according to the configuration information of the job, and generate a job running schedule.

[0107] The second module 603 is used to obtain key operation indicators of the job through the operation history data of the job, and generate a job operation history indicator table.

[0108] The third module 605 is used to obtain the completed jobs through the job running schedule to calculate and display the progress of application completion.

[0109] The fourth module 607 is used to display abnormal situations and drill down to the abnormal job information in combination with the job running history indicator table, and obtain the abnormal job list of the application.

[0110] According to an embodiment, the first module 601 is configured to: classify each scene according to the configuration information of the job; identify the target application associated with each scene as an end node application; identify the end node application of the scene, where one scene corresponds to one end node application; determine the dependency relationship between the end node application and the job of the corresponding scene; and calculate each node application step by step forward according to the dependency relationship between the jobs to obtain the AOE network of the scene, thereby generating the actual job list.

[0111] According to an embodiment, the second module 603 is configured to: poll the current completion status of the jobs according to the number of jobs contained in each application in the scenario, wherein the real-time progress of each application = the number of jobs that have been successfully executed / the total number of jobs contained in the application in the scenario; and use the progress of the end node application to represent the overall progress of the scenario.

[0112] According to an embodiment, the fourth module 607 is configured to: obtain abnormal situation classification; calculate the proportion of abnormal operations through the classification of the abnormal situation; and perform differentiated display according to the proportion of abnormal operations.

[0113] The batch processing monitoring device includes a processor and a memory. The first module 601, the second module 603, the third module 605 and the fourth module 607 are all stored in the memory as program units, and the processor executes the program modules stored in the memory to implement corresponding functions.

[0114] The beneficial effects of the batch processing monitoring device of the present application are the same as the batch processing monitoring method described above, which will not be elaborated here.

[0115] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the batch monitoring method is implemented by adjusting kernel parameters.

[0116] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0117] An embodiment of the present application provides a storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned batch processing monitoring method is implemented.

[0118] An embodiment of the present application provides a processor, which is used to run a program, wherein the batch processing monitoring method is executed when the program is running.

[0119] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store batch monitoring data. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a batch monitoring method is implemented.

[0120] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, the batch processing monitoring device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 7 The memory of the computer device can store various program modules constituting the batch processing monitoring device, such as: Figure 6 The first module, the second module, the third module and the fourth module shown in the figure. The computer program composed of each program module enables the processor to execute the steps of the batch monitoring method of each embodiment of the present application described in this specification.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0128] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0130] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A batch monitoring method, characterized in that: The method comprises: Get the scheduled running jobs of the application according to the job configuration information and generate a job running schedule, including: Identify the end node applications of the scenarios, where one scenario corresponds to one end node application; Determine the dependency between the end node application and the job of the corresponding scenario; According to the dependency relationship between the jobs, the node applications are calculated step by step to obtain the AOE network of the scene, thereby generating the actual job list. The step of identifying the end node application of the scenario includes: classifying each scenario according to the configuration information of the job; identifying the target application associated with each scenario as the end node application; Obtain key operation indicators of the job through the operation history data of the job, and generate a job operation history indicator table; Obtain the completed jobs through the job running schedule to calculate and display the progress of application completion; Combined with the job running history indicator table, display the abnormal situation and drill down to the abnormal job information to obtain the abnormal job list of the application, including: From the AOE network of the scene, get the path with the maximum path length and mark it as the critical path; According to the critical path, the abnormal situation is marked, including marking job failure, job timeout and job delayed start.

2. The method according to claim 1, characterized in that Acquiring the key operation index of the job through the operation history data of the job includes taking the median of the operation history data as the key operation index of the job.

3. The method according to claim 1 or 2, characterized in that: The key operating indicators of the operation include: The execution time of the job, the time when the job starts running, and the time when the job ends running.

4. The method according to claim 1, characterized in that: The method further includes: The abnormal job list is refreshed periodically, and the refreshed data is kept in a cache so as to directly read the refreshed data.

5. The method according to claim 1, characterized in that: Through the job running schedule, the jobs that have been completed are obtained to calculate and display the progress of each application, including: According to the number of jobs contained in each application in the scenario, poll the current completion status of the job, where the real-time progress of each application = the number of successfully executed jobs / the total number of jobs contained in the application in the scenario; The overall progress of the scene is represented by the progress of the terminal node application.

6. A batch processing monitoring device, characterized in that: The device comprises: The first module is used to obtain the planned running jobs of the application according to the configuration information of the job and generate a job running schedule; The second module is used to obtain key operation indicators of the job through the operation history data of the job and generate a job operation history indicator table; The third module is used to obtain the completed jobs through the job running schedule to calculate and display the progress of application completion; The fourth module is used to display abnormal situations and drill down to the abnormal job information in combination with the job running history indicator table, and obtain the abnormal job list of the application. The first module is configured as follows: Classify each scene according to the configuration information of the job; Identify the target application associated with each scenario as an end node application; Identify the end node applications of the scenarios, where one scenario corresponds to one end node application; Determine the dependency between the end node application and the job of the corresponding scenario; According to the dependency relationship between the jobs, the node applications are calculated step by step to obtain the AOE network of the scene, thereby generating the actual job list. The fourth module is configured as follows: From the AOE network of the scene, get the path with the maximum path length and mark it as the critical path; According to the critical path, the abnormal situation is marked, including marking job failure, job timeout and job delayed start.

7. The device according to claim 6, characterized in that The second module is configured as follows: According to the number of jobs contained in each application in the scenario, poll the current completion status of the job, where the real-time progress of each application = the number of successfully executed jobs / the total number of jobs contained in the application in the scenario; The progress of the terminal node application represents the overall progress of the scene.

8. The device according to claim 7, characterized in that The fourth module is further configured to: Get the abnormal situation classification; By classifying the abnormal situations, calculating the proportion of abnormal operations; Differentiate and display the abnormal operations according to the proportion of the abnormal operations.

9. A batch processing monitoring device, comprising a processor and a memory, characterized in that: The processor is configured to execute the method according to any one of claims 1-5.

10. A processor, characterized in that: The method is configured to execute the batch process monitoring method according to any one of claims 1 to 5.

11. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the processor is configured to perform the method according to any one of claims 1 to 5.

12. A computer program product, comprising a computer program, characterized in that The computer program implements the method according to any one of claims 1 to 5 when executed by a processor.

Images