Intelligent job scheduling method and device based on medical data blood relationship

Abstract

The invention provides an intelligent job scheduling method and device based on medical data blood relationship. The method comprises the following steps: acquiring configuration information of all warehouse counting jobs, and extracting metadata and a structured query language; analyzing and generating a table-level blood relationship according to the structured query language, and collecting all blood relationships to generate a directed acyclic data link diagram; according to the directed acyclic data link diagram and the operation information in the operation group specified by the user, regenerating the directed acyclic data link diagram of the operation which only needs to be executed, and converting and optimizing the directed acyclic data link diagram into an execution plan; according to the execution plan and the execution state and strategy configuration of the operation, optimal dynamic scheduling is carried out, and an operation instruction is generated; and executing the structured query language according to the operation instruction to complete data processing. According to the method, the blood relationship of the medical data can be automatically identified to generate the directed acyclic graph, job scheduling is automatically carried out, ordered execution of data jobs is ensured, the method has exception recovery and re-execution capabilities, and manpower does not need to participate in a large amount of job scheduling work.

Description

Technical Field

[0001] The present invention belongs to the field of medical information, and in particular relates to an intelligent job scheduling method and device based on medical data blood relationship. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] When building a medical data warehouse, you need to create data tables at the ODS, DWD, DI M, DWS, ADS and other layers, and complete data processing and writing through SQL or other data synchronization jobs. These jobs often have a strict execution order, and the data processing and writing of the upstream data table must be completed before the data operation of the downstream data table can be started.

[0004] The traditional scheduling method is to manually specify a timed task to trigger the data job, and to achieve orderly scheduling of the job through the trigger time. The inventors found that the traditional scheduling method has the following problems:

[0005] 1. The execution time of each job is not fixed. When the running time of the upstream job is longer than the interval between the upstream and downstream job trigger times, the calculation of the downstream data table will start before the calculation of the data in the upstream data table is completed, which will eventually lead to errors in the calculation of topics or indicators.

[0006] 2. Data warehouses usually have a large number of data tables, so more data jobs need to be orchestrated. When the number of data jobs increases, such as hundreds of tasks, it is difficult for operation and maintenance personnel to achieve orderly execution by controlling the trigger time.

[0007] 3. A large number of jobs will also generate a large number of blood relationships, which are difficult to sort out and maintain manually. Even if a big data scheduling platform with workflow scheduling is used, it is very difficult to manually maintain hundreds or thousands of node relationships.

[0008] 4. When a job fails, there is no recovery or re-run mechanism, and downstream jobs will still be executed on time, resulting in errors in the calculation of the final indicators. Manual intervention is required to ensure orderly execution. Summary of the invention

[0009] In order to solve the above problems, the present invention proposes an intelligent job scheduling method and device based on the blood relationship of medical data. The present invention can automatically identify the blood relationship of medical data to generate a directed acyclic graph DAG, and automatically schedule jobs based on DAG, strictly ensuring the orderly execution of data jobs, and having the ability to recover from exceptions and re-execute, without the need for manual participation in a large amount of job scheduling work.

[0010] According to some embodiments, the present invention adopts the following technical solutions:

[0011] An intelligent job scheduling method based on medical data blood relationship includes the following steps:

[0012] Obtain configuration information for all data warehouse jobs, extract metadata and structured query language;

[0013] Generate table-level kinship relationships based on structured query language analysis, and aggregate all kinship relationships to generate a directed acyclic data link graph;

[0014] Regenerate the directed acyclic data link graph of the jobs that need to be executed based on the directed acyclic data link graph and the job information in the job group specified by the user, and convert and optimize it into an execution plan;

[0015] Perform optimal dynamic scheduling and generate job instructions based on the execution plan, job execution status, and policy configuration;

[0016] Execute the structured query language according to the job instructions to complete data processing.

[0017] As an optional implementation method, the process of generating table-level blood relationships based on structured query language parsing includes: parsing relevant statements of the structured query language to obtain the source of data in the current data table, forming a single-table blood relationship analysis; parsing all structured query languages ​​to obtain blood relationships, and merging all blood relationships into a complete data link relationship.

[0018] As an optional implementation method, the process of regenerating a directed acyclic data link graph for jobs that only need to be executed based on the directed acyclic data link graph and the job information in the job group specified by the user includes: recalculating all predecessor nodes through the job group specified by the user, re-inferring a directed acyclic graph that only contains the current node and the predecessor node, generating an execution plan based on the directed acyclic graph, and then optimizing the execution plan according to the behavior strategy configured by the user.

[0019] As an optional implementation, the policy configuration includes job parallelism, whether to automatically retry, the number of retries and / or whether to skip abnormal jobs.

[0020] As an optional implementation, when a job is completed, its subsequent job is automatically scheduled for execution; if the job fails, the job is skipped or the scheduling is terminated according to the configured policy.

[0021] An intelligent job scheduling device based on medical data blood relationship, comprising:

[0022] The job configuration collection module is configured to obtain the configuration information of all data warehouse jobs and extract metadata and structured query language;

[0023] The blood relationship analysis module is configured to generate table-level blood relationships according to the structured query language analysis, and collect all blood relationships to generate a directed acyclic data link graph;

[0024] An execution plan optimization module is configured to regenerate a directed acyclic data link graph of jobs that need to be executed based on the directed acyclic data link graph and job information in a job group specified by a user, and convert and optimize it into an execution plan;

[0025] The job scheduling module is configured to perform optimal dynamic scheduling and generate job instructions according to the execution plan, the execution status and policy configuration of the job;

[0026] The job execution module is configured to execute the structured query language according to the job instructions to complete data processing.

[0027] As an optional implementation, the blood relationship analysis module is configured to parse the relevant statements of the structured query language to obtain the source of the data in the current data table, forming a single-table blood relationship analysis; parse all structured query languages ​​to obtain blood relationships, and merge all blood relationships into a complete data link relationship.

[0028] As an optional implementation, the execution plan optimization module is configured to recalculate all predecessor nodes through a job group specified by the user, re-infer a directed acyclic graph containing only the current node and the predecessor node, generate an execution plan based on the directed acyclic graph, and then optimize the execution plan according to the behavior strategy configured by the user.

[0029] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0030] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention realizes intelligent resource scheduling. The user only needs to add the jobs to be scheduled to the job group and configure the scheduling strategy. The device can automatically parse its upstream jobs and dynamically schedule the job execution according to the resource usage.

[0033] 2. The present invention realizes intelligent recovery from abnormalities. The device automatically maintains tens of thousands of nodes, strictly ensures the orderly execution of operations, and realizes abnormal recovery and link re-running.

[0034] 3. The present invention completely manages and executes big data operations, eliminating the cost of full-time personnel responsible for development and maintenance, thereby achieving the effect of reducing costs and increasing efficiency.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0037] Figure 1 This is a structural diagram of an intelligent job scheduling device based on medical data blood relationship in one embodiment;

[0038] Figure 2 The present invention is a flow chart of an intelligent job scheduling method based on medical data blood relationship according to an embodiment. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0042] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0043] Embodiment 1

[0044] An intelligent job scheduling device based on medical data blood relationship, such as Figure 1 As shown, it includes a job configuration collection module 101, a blood relationship analysis module 102, an execution plan optimization module 103, a job scheduling module 104, and a job execution module 105.

[0045] The job configuration collection module 101 collects the data warehouse metadata information and SQL job information from the data center as the basic data for the blood relationship analysis module.

[0046] Blood relationship analysis module 102: By using Ali ibaba Druid to parse the INSERT INTO or CREATE VIEW statement in a SQL job, the single-table blood relationship is obtained. The steps are as follows: first, the SQL is parsed into the abstract syntax tree of SELECT and INSERT through the parseStatement of SQLUtils, and the abstract syntax tree is traversed using SchemaStatVis itor to obtain all query tables as source tables, and a one-to-many relationship between the target table and the source table is constructed to form a single-table blood relationship. Then, the above-mentioned parsing steps are performed on the SQL statements in all SQL jobs to obtain the blood relationship of each table, and all blood relationships are uniformly stored in the data structure of the predecessor node and the successor node to form a directed acyclic graph DAG, which is a complete data link relationship. By traversing the DAG path, the correct job execution order of the SQL jobs corresponding to each table can be obtained.

[0047] Execution plan optimization module 103: uses the SQL job in the job group specified by the user as the successor node, traverses and searches for all its predecessor nodes in the DAG in the above step, recomposes the DAG containing only the current node and the predecessor node, generates an execution plan according to the DAG graph, and the data structure of the execution plan is a binary tuple list. Then, according to the user-configured behavior strategy such as predecessor first or depth first, the binary tuple list is sorted to generate an optimized execution plan.

[0048] Job scheduling module 104: Responsible for dynamically deciding the scheduling and execution of jobs according to the optimized execution plan. Its decision is affected by the configuration executed by the user, such as job parallelism, whether to automatically retry, the number of retries, whether to skip abnormal jobs, etc. Its dynamic decision is realized as the job scheduling module will monitor the execution of jobs in real time. When the job parallelism is greater than 1, the job scheduling module will give priority to running jobs without predecessor nodes or completed predecessor nodes to achieve the simultaneous running of multiple jobs without affecting the correct execution order of the jobs; when the job is completed, its successor jobs will be automatically scheduled for execution; if the job fails, the job can be retried, skipped, or the scheduling can be terminated according to the strategy. If the automatic recovery after the job fails still fails, the failed job and all its successor jobs can be rescheduled through the abnormal restart function after the abnormal job is manually resolved.

[0049] Job execution module 105: responsible for submitting and stopping jobs, and obtaining job execution status in real time. It supports plug-in extensions, such as Flink tasks, SQL tasks, Datax tasks, etc.

[0050] Embodiment 2

[0051] like Figure 2 As shown, based on the device of embodiment 1, a DAG-based medical data lineage intelligent generation method includes:

[0052] The job configuration collection module obtains the configuration information of all data warehouse jobs from the data center, extracts metadata and SQL, and outputs them to the blood relationship analysis module;

[0053] The blood relationship analysis module generates table-level blood relationships based on SQL analysis, collects all blood relationships to generate a DAG data link diagram, and outputs it to the execution plan optimization module;

[0054] The execution plan optimization module regenerates the DAG data link diagram of the jobs that need to be executed based on the DAG data link diagram and the job information in the job group specified by the user, and converts and optimizes it into an execution plan and outputs it to the job scheduling module;

[0055] The job scheduling module performs optimal dynamic scheduling based on the execution plan, job execution status, and policy configuration, and calls the job execution module to trigger the job;

[0056] The job execution module executes the corresponding data tasks on the data platform according to the job configuration, and obtains the execution status in real time and feeds it back to the job scheduling module.

[0057] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0059] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. An intelligent job scheduling method based on medical data blood relationship, characterized in that: The following steps are involved: Obtain configuration information for all data warehouse jobs, extract metadata and structured query language; Generate table-level kinship relationships based on structured query language analysis, and aggregate all kinship relationships to generate a directed acyclic data link graph; Regenerate the directed acyclic data link graph of the jobs that need to be executed based on the directed acyclic data link graph and the job information in the job group specified by the user, and convert and optimize it into an execution plan; Perform optimal dynamic scheduling and generate job instructions based on the execution plan, job execution status, and policy configuration; Execute the structured query language according to the job instructions to complete data processing.

2. The intelligent job scheduling method based on medical data blood relationship as claimed in claim 1, characterized in that: The process of generating table-level blood relations according to structured query language analysis includes: parsing the relevant statements of the structured query language to find out the source of data in the current data table, forming a single-table blood relation analysis; parsing all structured query languages ​​to obtain blood relations, and merging all blood relations into a complete data link relationship.

3. The intelligent job scheduling method based on medical data blood relationship as claimed in claim 1, characterized in that: The process of regenerating the directed acyclic data link graph of the jobs that need to be executed only based on the directed acyclic data link graph and the job information in the job group specified by the user includes: recalculating all predecessor nodes through the job group specified by the user, re-inferring the directed acyclic graph that only includes the current node and the predecessor node, generating an execution plan based on the directed acyclic graph, and then optimizing the execution plan according to the behavior strategy configured by the user.

4. The intelligent job scheduling method based on medical data blood relationship as claimed in claim 1, characterized in that: The policy configuration includes job parallelism, whether to automatically retry, the number of retries, and / or whether to skip abnormal jobs.

5. The intelligent job scheduling method based on medical data blood relationship as claimed in claim 1, characterized in that: When a job is completed, its successor jobs are automatically scheduled for execution. If a job fails, the job is skipped or the scheduling is terminated according to the configured strategy.

6. An intelligent job scheduling device based on medical data blood relationship, characterized in that: include: The job configuration collection module is configured to obtain the configuration information of all data warehouse jobs and extract metadata and structured query language; The blood relationship analysis module is configured to generate table-level blood relationships according to the structured query language analysis, and collect all blood relationships to generate a directed acyclic data link graph; An execution plan optimization module is configured to regenerate a directed acyclic data link graph of jobs that need to be executed based on the directed acyclic data link graph and job information in a job group specified by a user, and convert and optimize it into an execution plan; The job scheduling module is configured to perform optimal dynamic scheduling and generate job instructions according to the execution plan, the execution status and policy configuration of the job; The job execution module is configured to execute the structured query language according to the job instructions to complete data processing.

7. The intelligent job scheduling device based on medical data blood relationship as claimed in claim 6, characterized in that: The blood relationship analysis module is configured to parse the relevant statements of the structured query language to find out the source of the data in the current data table, forming a single-table blood relationship analysis; parse all structured query languages ​​to obtain blood relationships, and merge all blood relationships into a complete data link relationship.

8. The intelligent job scheduling device based on medical data blood relationship as claimed in claim 6, characterized in that: The execution plan optimization module is configured to recalculate all predecessor nodes through the job group specified by the user, re-infer a directed acyclic graph containing only the current node and the predecessor node, generate an execution plan based on the directed acyclic graph, and then optimize the execution plan according to the behavior strategy configured by the user.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 5.

10. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 5 are completed.

Images