Operation management method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses a job management method, device and equipment, a medium and a product, and relates to the technical field of financial science and technology. The method comprises the following steps: in response to an execution instruction of each job of the distributed batch job system, generating a job instance of each job, and determining dependency information of each job based on each job instance; generating a dependency relationship graph among the jobs based on the job instances and the dependency information; and in response to an execution exception instruction of a target job, determining reference jobs associated with the target job based on the dependency graph, and determining an exception reason of the target job based on the job state of each reference job. According to the scheme provided by the embodiment of the invention, the abnormal reason of each operation in the distributed batch operation system can be quickly and accurately determined, the execution efficiency of an algorithm is improved, and help is provided for stable operation of the distributed batch operation system.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to an operation management method, device, equipment, medium and product. Background Art

[0002] A distributed batch job system is a distributed computing platform designed specifically for processing large-scale, periodic, or one-time batch tasks. It is suitable for common business needs such as "end-of-day batch job processing," "monthly settlement statistics," and "risk calculation" in scenarios such as finance, banking, insurance, telecommunications, and big data analysis.

[0003] Distributed batch jobs may become unstable in a distributed environment, so the execution results of the jobs need to be monitored and observed. How to quickly and accurately determine the cause of abnormalities in each job is a key research issue in the industry. Summary of the Invention

[0004] The present invention provides a job management method, apparatus, device, medium and product to quickly and accurately determine the abnormal causes of each job in a distributed batch job system, improve the execution efficiency of the algorithm, and provide assistance for the stable operation of the distributed batch job system.

[0005] According to one aspect of the present invention, there is provided a job management method, the method comprising:

[0006] In response to execution instructions of each job of the distributed batch job system, generating a job instance of each job, and determining dependency information of each job based on each job instance;

[0007] Generate a dependency graph between the jobs based on the job instances and the dependency information;

[0008] In response to an execution exception instruction of a target job, reference jobs associated with the target job are determined based on the dependency graph, and a cause of the exception of the target job is determined based on a job status of each of the reference jobs.

[0009] According to another aspect of the present invention, there is provided a job management device, the device comprising:

[0010] a dependency information determination module, configured to generate a job instance of each job in response to an execution instruction of each job in the distributed batch job system, and determine dependency information of each job based on each job instance;

[0011] A dependency graph generating module, configured to generate a dependency graph between jobs based on the job instances and the dependency information;

[0012] The abnormality cause determination module is used to determine, in response to an execution abnormality instruction of a target job, reference jobs associated with the target job based on the dependency graph, and determine an abnormality cause of the target job based on a job status of each reference job.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the job management method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the job management method described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the job management method according to any embodiment of the present invention.

[0019] The technical solution of the embodiment of the present invention generates a job instance of each job in response to the execution instruction of each job in the distributed batch job system, and determines the dependency information of each job based on each job instance; generates a dependency graph between each job based on each job instance and each dependency information; can determine the dependency information of each job in real time during the execution process of each job, and can draw a dependency graph based on each dependency information; responds to the execution exception instruction of the target job, determines the reference job associated with the target job based on the dependency graph, and determines the exception cause of the target job based on the job status of each reference job, and can quickly and accurately determine the exception cause of each job in the distributed batch job system based on the dependency graph between each job, thereby improving the execution efficiency of the algorithm and providing assistance for stable operation of the distributed batch job system.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flowchart of a job management method provided according to the first embodiment of the present invention;

[0023] Figure 2 A job dependency graph provided according to the first embodiment of the present invention;

[0024] Figure 3 This is a flowchart of a job management method provided according to the second embodiment of the present invention;

[0025] Figure 4 This is a flowchart of a job management method provided according to Embodiment 3 of the present invention;

[0026] Figure 5 This is a block diagram showing the execution status of each session of each job according to the third embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of a process for generating session-dependent data according to the third embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of a job management device provided according to a fourth embodiment of the present invention;

[0029] Figure 8 It is a structural diagram of an electronic device for implementing the job management method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a job management method provided according to the first embodiment of the present invention. This embodiment is applicable to monitoring the status of each job in a distributed batch job system. The method can be executed by a job management device, which can be implemented in the form of hardware and / or software. The job management device can be configured in electronic devices such as computers, servers, or tablet computers. Figure 1 As shown, the method includes:

[0034] Step 110 : In response to the execution instructions of each job in the distributed batch job system, generate a job instance of each job, and determine dependency information of each job based on each job instance.

[0035] Among them, the distributed batch job system can be an operation system used to process massive amounts of data in industries such as banking, insurance, securities, and e-commerce; for example, it can perform operations such as account settlement, interest calculation, or report generation every night; it can also perform operations such as transaction reconciliation and risk control checks after the daily closing.

[0036] It can be understood that in this embodiment, the distributed batch job system can perform multiple jobs, such as account settlement, interest calculation, report generation, transaction reconciliation or risk control inspection involved in the above examples; at the same time, each job may need to be performed multiple times, and in this embodiment, executing a certain job once is defined as a session.

[0037] In this embodiment, the dependency information of each job is the execution pipeline between jobs. For example, if the prerequisite for job B to start execution is that job A is completed, then job B depends on job A.

[0038] Optionally, in this embodiment, after receiving execution instructions for individual jobs in the distributed batch job system, job instances matching each job may be further generated. A job instance for a job may be a record of completing the job, including, for example, the start time, execution time, job status, and input and output data. It is understood that, in an optional implementation of this embodiment, a job instance may be generated for each session of any job.

[0039] Furthermore, the dependency information of each job can be determined based on the generated job instances; in this embodiment, the association relationship between each job instance can be established based on the start time or input and output data of each job instance, so as to determine the dependency information of each job; for example, if the output data of the first session is the input data of the second session, and the start time of the second session is later than the start time of the first session, then it can be determined that the first session is the upstream session of the second session, that is, the second session is the downstream session of the first session, that is, the second session needs to rely on the execution result of the first session to execute normally.

[0040] Step 120: Generate a dependency graph between the jobs based on the job instances and the dependency information.

[0041] Optionally, in this embodiment, after each job instance is generated and the dependency information of each job is determined, a dependency graph between jobs may be further generated based on each job instance and the dependency information between jobs.

[0042] In an optional implementation of this embodiment, a dependency relationship between jobs can be established based on the dependency relationship between sessions. For example, in the above example, if the first session belongs to job a and the second session belongs to job b, then job b depends on job a.

[0043] Optionally, in this embodiment, after determining the dependency relationships of all jobs based on the information of each job instance, a dependency graph between jobs can be drawn based on the front-end drawing component. For example, Figure 2 This is a job dependency diagram provided according to the first embodiment of the present invention. Figure 2 As can be seen from the figure, job a depends on job S; job b depends on job a, job S, and file 1; job c depends on job a, job S, file 1, and job b; job E depends on job a, job S, file 1, job b, and job c.

[0044] Step 130 : In response to the execution exception instruction of the target job, determine reference jobs associated with the target job based on the dependency graph, and determine the exception cause of the target job based on the job status of each reference job.

[0045] The target job may be any job completed in the distributed job system, which is not limited in this embodiment.

[0046] In this embodiment, when it is determined that the target job is executed abnormally, for example, the execution result is wrong, the execution is delayed, or it is executed repeatedly, the reference job associated with the target job can be determined through the dependency graph generated by the above steps; for example, Figure 2 As shown, if the target job is job a, then the reference job is job S; if the target job is job E, then the reference jobs are job a, job S, file 1, job b, and job c.

[0047] Furthermore, after determining the reference jobs associated with the target job, the cause of the abnormality of the target job can be determined based on the job status of each reference job; for example, if the execution status of job S is abnormal, then the cause of the abnormality of job a is the abnormal execution of job S; if the execution status of job S is normal, then the cause of the abnormality of job a is unrelated to job S, and its abnormal cause is its own reason. The status of the executor that executes job a can be further queried to further determine the cause of the abnormality of job a.

[0048] The technical solution of this embodiment generates a job instance of each job in response to the execution instruction of each job in the distributed batch job system, and determines the dependency information of each job based on each job instance; generates a dependency graph between each job based on each job instance and each dependency information; can determine the dependency information of each job in real time during the execution process of each job, and can draw a dependency graph based on each dependency information; in response to the execution exception instruction of the target job, determines the reference job associated with the target job based on the dependency graph, and determines the exception cause of the target job based on the job status of each reference job. Based on the dependency graph between each job, the exception cause of each job in the distributed batch job system can be quickly and accurately determined, thereby improving the execution efficiency of the algorithm and providing assistance for stable operation of the distributed batch job system.

[0049] Example 2

[0050] Figure 3 This is a flowchart of a job management method provided according to the second embodiment of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. Figure 3 As shown, the method includes:

[0051] Step 310: In response to the execution instructions of each job in the distributed batch job system, generate a job instance of each job.

[0052] Optionally, in this embodiment, in response to the execution instructions of each job of the distributed batch job system, generating a job instance of each job may include: parsing the execution instructions to obtain the execution plan of each job; obtaining the configuration file of the target job from the target database to obtain the configuration information of the target job; and generating a job instance of the target job based on the configuration information of the target job and the execution plan of the target job.

[0053] Among them, the target job includes at least two sessions, that is, the target job can be executed multiple times; it can be understood that there may be dependencies between the sessions of different jobs; for example, if the 0th session execution of job B depends on the 1st session execution of job A, then it means that the job has cross-session dependencies.

[0054] In this embodiment, the target database may be any database in the distributed batch job system, which is not limited in this embodiment; the target job may be one or more jobs matching the job execution instruction, which is not limited in this embodiment.

[0055] In an optional implementation of this embodiment, after receiving the execution instructions of each job in the distributed system, the execution instructions can be parsed first to obtain the execution plan of each job, such as the execution time, execution venue or execution purpose of each job, which is not limited in this embodiment.

[0056] At the same time, the configuration file of the target job can also be obtained from the target database, and then the configuration information of the target job can be obtained from the configuration file; illustratively, the configuration information of the target job may include: job name, resource requirements, parameter settings, etc.

[0057] Furthermore, a job instance of the target job can be generated based on the acquired configuration information of the target job and the execution plan of the target job; further, after the job instance of the target job is generated, the job instance can be stored in the target database for subsequent use.

[0058] In an optional implementation of this embodiment, after obtaining the configuration information of the target job and the execution plan of the target job, the obtained configuration information of the target job and the execution plan of the target job can be filled into a preset job instance template or class to obtain a job instance of the target job; it can be understood that the job instance of the target job is an execution record of the distributed batch job system executing the target job once.

[0059] The advantage of this setting is that the job instance of each job can be quickly obtained, which helps to generate the dependency graph between the jobs later.

[0060] Step 320: Determine dependency information of each job based on each job instance.

[0061] Optionally, in this embodiment, after generating the job instance of each job, the dependency information between the jobs can be further determined based on the generated job instances; in this embodiment, determining the dependency information of each job based on each of the job instances can include: obtaining the first job instance information of the first job instance, and determining a first data table in the target database based on the first job instance information; obtaining each session of the first job instance and the session dependency information between the sessions from the first data table; and determining the dependency relationship between the first job and other jobs based on the session dependency information.

[0062] Among them, the first job matches the first job instance; the first job can be any job, or it can be the target job involved in the above steps. In this embodiment, it is named the first job only for the convenience of description, which is not a limitation of this embodiment.

[0063] In an optional implementation of this embodiment, after generating a job instance for each job and storing each job instance in a target database, first job instance information of the first job instance can be further obtained, and based on the first job instance information, a first data table matching the first job instance can be queried in the target database; wherein, the first data table stores instance information of the first job instance.

[0064] Furthermore, the various sessions of the first job instance and the dependency relationships between the sessions can be obtained from the first data table; wherein, the various sessions of the first job instance can depend on the sessions of other job instances; based on this, the dependency relationships between the first job and other jobs can be determined through the dependency information of each session.

[0065] For example, if a first session of a first job instance depends on a second session of a second job instance, then it can be determined that the first job depends on the second job.

[0066] The advantage of this setting is that the dependency between different jobs can be quickly determined based on the dependency between each session, which improves the accuracy of the algorithm and solves the problem of difficulty in determining the dependency between jobs based only on job information.

[0067] Step 330: The front-end component draws a graph based on the session dependency information of each session of each job instance and the dependency information between jobs to obtain the dependency graph.

[0068] Optionally, in this embodiment, after the dependency relationships between the jobs are determined, a dependency graph may be drawn based on the dependency relationships between the jobs based on a front-end component of the distributed batch job system, for example, a drawing component.

[0069] The advantage of this setting is that you can quickly obtain a dependency graph, which helps to subsequently determine the cause of the abnormal job.

[0070] Step 340 : In response to the execution exception instruction of the target job, determine reference jobs associated with the target job based on the dependency graph, and determine the cause of the exception of the target job based on the job status of each reference job.

[0071] Optionally, in this embodiment, in response to an execution exception instruction of the target job, a reference job associated with the target job is determined based on the dependency graph, and the cause of the exception of the target job is determined based on the job status of each of the reference jobs. This may include: when it is determined that the target job is abnormal, determining the first upstream job of the target job based on the dependency graph; if it is determined that the execution result of the first upstream job is normal, determining that the cause of the exception of the target job is the target job's own cause; if it is determined that the execution result of the first upstream job is abnormal, and the execution result of the second upstream job of the first upstream job is normal, determining that the cause of the exception of the target job is the abnormality of the first upstream job.

[0072] In an optional implementation of this embodiment, when it is determined that the target job is abnormal, the first upstream job of the target job can be determined based on the dependency graph drawn in the above steps; wherein, the first upstream job of the target job can be one or all upstream jobs of the target job, which is not limited in this embodiment; for example, Figure 2 As shown, if the target job is job a, then the first upstream job can be job S; if the target job is job E, then the first upstream job can be job c; it can also be job c and job b; it can also be job S, job a, job b and job c.

[0073] In an optional implementation of this embodiment, if it is determined that the execution result of the first upstream job of the target job is normal, then the cause of the abnormality of the target job can be determined to be the target job itself. The execution log of the target job can be further checked to accurately determine the cause of the abnormality of the target job.

[0074] In another optional implementation of this embodiment, if it is determined that the execution result of the first upstream job is abnormal, and the execution result of the second upstream job of the first upstream job (that is, the upstream job of the first upstream job) is normal, then it can be determined that the cause of the abnormality of the target job is the abnormality of the first upstream job.

[0075] The solution of this embodiment, when it is determined that the target job is abnormal, determines the first upstream job of the target job based on the dependency graph; if it is determined that the execution result of the first upstream job is normal, determines that the cause of the abnormality of the target job is the target job itself; if it is determined that the execution result of the first upstream job is abnormal, and the execution result of the second upstream job of the first upstream job is normal, determines that the cause of the abnormality of the target job is the abnormality of the first upstream job. This can quickly determine the cause of the abnormality of the target job, providing assistance in resolving the job abnormality.

[0076] Example 3

[0077] Figure 4 This is a flowchart of a job management method provided according to the second embodiment of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. Figure 4 As shown, the method includes:

[0078] Step 410: In response to the execution instructions of each job in the distributed batch job system, generate a job instance of each job.

[0079] Step 420: Determine dependency information of each job based on each job instance.

[0080] Step 430: Generate a dependency graph between the jobs based on the job instances and the dependency information.

[0081] Step 440: Obtain the execution results of each session of each job from the session record table, and establish a mapping relationship between each execution result and color; create a block diagram based on the target library, and determine the color of each block based on the mapping relationship between each execution result and color; in response to a click instruction of the target block, query the information of the session corresponding to the target block, determine the upstream session and downstream session associated with the target session, and highlight the upstream session and the downstream session; update the dependency graph based on the upstream session and / or the downstream session.

[0082] Wherein, one box in the block diagram represents one session; wherein, the session record table records the execution results of all sessions of all jobs in the distributed batch job system.

[0083] Optionally, in this embodiment, after drawing the dependency graph between the jobs, the direct results of each session of each job can be obtained from the session record table, and a mapping relationship between each execution result and color can be established; for example, normal corresponds to green, abnormal corresponds to red, delayed execution corresponds to yellow, or not executed corresponds to gray, etc.

[0084] Furthermore, a block diagram can be created based on the target library, and the color of each block can be determined based on the mapping relationship between each execution result and the color; wherein a box in the block diagram represents a session; for example, Figure 5 This is a block diagram of the execution status of each session of each job provided according to the third embodiment of the present invention, wherein job B includes 16 sessions, job C includes 16 sessions, and job E includes one session.

[0085] Furthermore, if a click instruction of a target block (any session) is received, the information of the session corresponding to the target block can be queried, and the upstream sessions and downstream sessions associated with the target session can be determined, and the upstream sessions and downstream sessions can be highlighted. Figure 5 In the figure, session 5 of assignment b and session 5 of assignment c are highlighted.

[0086] Furthermore, the dependency graph determined above can be updated based on the upstream session, downstream session, or both upstream and downstream sessions of the target session. For example, if the upstream session of the target session of job X is determined to be session 5 of job Y, and a dependency relationship between job X and job Y is not established in the dependency graph, then a dependency relationship between job X and job Y can be established based on this, i.e., the dependency graph can be updated.

[0087] It is understood that in this embodiment, the session record table maintains the job definition. Therefore, through the job definition and job relationship, upstream and downstream jobs and sessions can be identified. For example, after selecting a job instance, the session record table can be used to query the job representation of the corresponding session information. The session record table can also be used to obtain the upstream and downstream jobs corresponding to the job identification. Once the upstream and downstream are obtained, they can be displayed in the job dependency graph.

[0088] The advantage of this setting is that you can observe the execution results of each session more intuitively, establish the association relationship of the remaining dependency graphs, and quickly determine the relationship between jobs based on the sessions.

[0089] Step 450: Click the box corresponding to the abnormal scene matching the target job in the block diagram to obtain the first highlighted result in the block diagram; determine the first upstream scene and the first downstream scene corresponding to the target scene based on the first highlighted result; determine the abnormal cause of the target scene based on the color information of the first upstream scene and / or the first downstream scene.

[0090] In this embodiment, when it is determined that the target job is abnormal, the block corresponding to the abnormal scene of the target job can be clicked in the created block diagram to obtain the first highlighted result in the block diagram; further, the first upstream scene and the first downstream scene corresponding to the target scene can be determined based on the first highlighted result; the abnormal cause of the target scene can be determined based on the color information of the first upstream scene and / or the first downstream scene.

[0091] In one example of this embodiment, Figure 5 In the example, if session 5 of job c is the target session, its upstream sessions are session 5 of job b. It can be understood that if session 5 of job b is executed abnormally, then session 5 of job c is also abnormal.

[0092] The solution of this embodiment is to click the box corresponding to the abnormal scene matching the target job in the block diagram to obtain the first highlighted result in the block diagram; determine the first upstream scene and the first downstream scene corresponding to the target scene based on the first highlighted result; determine the abnormal cause of the target scene based on the color information of the first upstream scene and / or the first downstream scene, so as to quickly determine the abnormal cause of the target job, and provide help to improve the stability of the distributed batch job system.

[0093] In order to better understand the embodiments of the present invention, Figure 6 This is a schematic diagram of a process for generating session-dependent data according to Embodiment 3 of the present invention. The process mainly includes two processes: an online instance generation process and an offline monitoring diagram generation process. The online instance generation process may include:

[0094] Step 610: Refresh the job instance;

[0095] Step 611: Read job configuration information;

[0096] Step 612: Generate a job instance (session) based on the job information and the execution plan;

[0097] Step 613: Insert the job instance into the database;

[0098] The offline generation process of monitoring charts may include:

[0099] Step 620: Read the job instance information (session);

[0100] Step 621: Read job dependency information.

[0101] Step 622: Generate a dependency graph between jobs and sessions.

[0102] Step 623: Store the relationship diagram data into the database.

[0103] In this embodiment, based on the online instance generation process, the instance generation data can be parsed asynchronously, decoupling from the online generation process. The asynchronous processing method can obtain data change information through timed task refreshes, monitoring database information, and monitoring instance validation messages.

[0104] The solution of the embodiment of the present invention clearly presents the execution status and dependency relationships of multiple sessions in a graphical manner through a structured display method.

[0105] Example 4

[0106] Figure 7 Schematic diagram of a structure of a job management device provided according to the fourth embodiment of the present invention. Figure 7 As shown, the apparatus includes: a dependency information determination module 710 , a dependency graph generation module 720 and an abnormality cause determination module 730 .

[0107] The dependency information determination module 710 is configured to generate a job instance of each job in response to an execution instruction of each job in the distributed batch job system, and determine dependency information of each job based on each job instance;

[0108] A dependency graph generating module 720 is configured to generate a dependency graph between jobs based on the job instances and the dependency information.

[0109] The abnormality cause determination module 730 is configured to determine, in response to an execution abnormality instruction of a target job, reference jobs associated with the target job based on the dependency graph, and determine an abnormality cause of the target job based on job statuses of the reference jobs.

[0110] The solution of this embodiment is to generate a job instance of each job in response to the execution instruction of each job in the distributed batch job system through a dependency information determination module, and determine the dependency information of each job based on each job instance; generate a dependency graph between each job based on each job instance and each dependency information through a dependency graph generation module; respond to the execution exception instruction of the target job through an exception cause determination module, determine the reference job associated with the target job based on the dependency graph, and determine the exception cause of the target job based on the job status of each reference job. This can quickly and accurately determine the exception cause of each job in the distributed batch job system, improve the execution efficiency of the algorithm, and provide assistance for stable operation of the distributed batch job system.

[0111] In an optional implementation of this embodiment, the dependency information determination module 710 is specifically configured to:

[0112] Parsing the execution instructions to obtain an execution plan for each of the jobs;

[0113] Obtaining a configuration file of a target job from a target database to obtain configuration information of the target job;

[0114] Generate a job instance of the target job based on the configuration information of the target job and the execution plan of the target job;

[0115] The target operation includes at least two sessions.

[0116] In an optional implementation of this embodiment, the dependency information determining module 710 is further specifically configured to:

[0117] Acquire first job instance information of the first job instance, and determine a first data table in a target database based on the first job instance information;

[0118] Acquire each session of the first job instance and session dependency information between the sessions from the first data table;

[0119] Determine a dependency relationship between a first job and other jobs based on dependency information of each session; wherein the first job matches the first job instance.

[0120] In an optional implementation of this embodiment, the dependency graph generation module 720 is specifically used to: draw a graph based on the session dependency information of each session of each job instance and the dependency information between each job through the front-end component to obtain the dependency graph.

[0121] In an optional implementation of this embodiment, the abnormality cause determination module 730 is specifically configured to:

[0122] If it is determined that the target job is abnormal, determining a first upstream job of the target job based on the dependency graph;

[0123] If it is determined that the execution result of the first upstream job is normal, determining that the abnormal cause of the target job is a problem of the target job itself;

[0124] If it is determined that the execution result of the first upstream job is abnormal, and the execution result of the second upstream job of the first upstream job is normal, then it is determined that the abnormal cause of the target job is the abnormality of the first upstream job.

[0125] In an optional implementation of this embodiment, the job management device further includes: a block diagram creation module, configured to:

[0126] Obtain the execution results of each session of each job from the session record table, and establish a mapping relationship between each execution result and color;

[0127] Creating a block diagram based on the target library, and determining the color of each block based on a mapping relationship between each execution result and the color; wherein each block in the block diagram represents a session;

[0128] In response to a click instruction of a target block, query information of a session corresponding to the target block, determine upstream sessions and downstream sessions associated with the target session, and highlight the upstream sessions and the downstream sessions;

[0129] The dependency graph is updated based on the upstream session and / or the downstream session.

[0130] In an optional implementation of this embodiment, the abnormality cause determination module 730 is further specifically configured to:

[0131] Clicking a box corresponding to an abnormal session matching the target job in the block diagram to obtain a first highlighted result in the block diagram;

[0132] Determining a first upstream session and a first downstream session corresponding to the target session based on the first highlighted display result;

[0133] The abnormality cause of the target session is determined based on the color information of the first upstream session and / or the first downstream session.

[0134] The job management device provided by the embodiment of the present invention can execute the job management method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0135] In the technical solutions of the embodiments of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of the operation information involved (such as execution results, configuration information, etc.) comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0136] Example 5

[0137] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0138] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0139] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0140] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a job management method, which may include: in response to the execution instructions of each job of the distributed batch job system, generating a job instance of each of the jobs, and determining the dependency information of each job based on each of the job instances; generating a dependency graph between each job based on each of the job instances and each of the dependency information; in response to the execution exception instruction of the target job, determining a reference job associated with the target job based on the dependency graph, and determining the abnormal cause of the target job based on the job status of each of the reference jobs.

[0141] In some embodiments, the job management method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the job management method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the job management method in any other suitable manner (e.g., via firmware).

[0142] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0147] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0148] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0149] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

[0150] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the database detection method provided in any embodiment of the present application.

[0151] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0153] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for job management, characterized in that: include: In response to execution instructions of each job of the distributed batch job system, generating a job instance of each job, and determining dependency information of each job based on each job instance; Generate a dependency graph between the jobs based on the job instances and the dependency information; In response to an execution exception instruction of a target job, reference jobs associated with the target job are determined based on the dependency graph, and a cause of the exception of the target job is determined based on a job status of each of the reference jobs.

2. The operation management method according to claim 1, characterized in that: The step of generating a job instance of each job in response to an execution instruction of each job of the distributed batch job system includes: Parsing the execution instructions to obtain an execution plan for each of the jobs; Obtaining a configuration file of a target job from a target database to obtain configuration information of the target job; Generate a job instance of the target job based on the configuration information of the target job and the execution plan of the target job; The target operation includes at least two sessions.

3. The operation management method according to claim 2, characterized in that: The determining of dependency information of each job based on each job instance includes: Acquire first job instance information of the first job instance, and determine a first data table in a target database based on the first job instance information; Acquire each session of the first job instance and session dependency information between the sessions from the first data table; Determine a dependency relationship between a first job and other jobs based on dependency information of each session; wherein the first job matches the first job instance.

4. The operation management method according to claim 3, characterized in that: The step of generating a dependency graph between jobs based on the job instances and the dependency information includes: The dependency graph is obtained by drawing a graph based on the session dependency information of each session of each job instance and the dependency information between jobs through the front-end component.

5. The operation management method according to claim 1, characterized in that: The step of responding to the execution exception instruction of the target job, determining reference jobs associated with the target job based on the dependency graph, and determining the cause of the exception of the target job based on the job status of each of the reference jobs, includes: If it is determined that the target job is abnormal, determining a first upstream job of the target job based on the dependency graph; If it is determined that the execution result of the first upstream job is normal, determining that the abnormal cause of the target job is a problem of the target job itself; If it is determined that the execution result of the first upstream job is abnormal, and the execution result of the second upstream job of the first upstream job is normal, then it is determined that the abnormal cause of the target job is the abnormality of the first upstream job.

6. The operation management method according to claim 4, characterized in that: The method further comprises: Obtain the execution results of each session of each job from the session record table, and establish a mapping relationship between each execution result and color; Creating a block diagram based on the target library, and determining the color of each block based on a mapping relationship between each execution result and the color; wherein each block in the block diagram represents a session; In response to a click instruction of a target block, query information of a session corresponding to the target block, determine upstream sessions and downstream sessions associated with the target session, and highlight the upstream sessions and the downstream sessions; The dependency graph is updated based on the upstream session and / or the downstream session.

7. The operation management method according to claim 6, characterized in that: The step of responding to the execution exception instruction of the target job, determining reference jobs associated with the target job based on the dependency graph, and determining the cause of the exception of the target job based on the job status of each of the reference jobs, includes: Clicking a box corresponding to an abnormal session matching the target job in the block diagram to obtain a first highlighted result in the block diagram; Determining a first upstream session and a first downstream session corresponding to the target session based on the first highlighted display result; The abnormality cause of the target session is determined based on the color information of the first upstream session and / or the first downstream session.

8. A work management device, characterized in that: include: a dependency information determination module, configured to generate a job instance of each job in response to an execution instruction of each job in the distributed batch job system, and determine dependency information of each job based on each job instance; A dependency graph generating module, configured to generate a dependency graph between jobs based on the job instances and the dependency information; The abnormality cause determination module is used to determine, in response to an execution abnormality instruction of a target job, reference jobs associated with the target job based on the dependency graph, and determine an abnormality cause of the target job based on a job status of each reference job.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the job management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the job management method according to any one of claims 1 to 7 when executed.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the job management method according to any one of claims 1 to 7.

Images