Method and device for scheduling data processing task, electronic equipment and storage medium

Abstract

The application discloses a kind of data processing task scheduling method and device, electronic equipment, storage medium, it is related to big data processing technical field or other related fields, wherein, the method comprises: obtaining data processing mapping, data processing mapping is the mapping configuration file obtained after being configured to preset template based on data processing demand;Data processing script is generated based on data processing mapping, and N data processing tasks are generated based on data processing script, N is positive integer;N data processing tasks are used to build scheduling flow chart based on, and scheduling flow chart is the directed acyclic graph for recording the execution order and dependent relationship of data processing task;N data processing tasks and scheduling flow chart are uploaded to batch scheduling system, and task scheduling is executed by batch scheduling system.The present application solves the technical problems that ETL task management scale is huge in the related art, and the efficiency of manually developing scheduling tasks is too low.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology or other related fields. Specifically, it relates to a method and apparatus for scheduling data processing tasks, an electronic device, and a storage medium. Background Technology

[0002] In the era of big data, data processing tasks (ETL tasks, Extract-Transform-Load) are typically diverse and may require collaborative processing across multiple nodes, such as machine learning tasks and database query tasks. Therefore, data processing tasks are generally scheduled by a scheduling system to improve the efficiency and accuracy of task scheduling.

[0003] The processing of big data tasks typically involves a large amount of data, which needs to be copied from storage nodes to task execution nodes. After the task is completed, the results are written to the storage nodes. Therefore, the data processing task involves reading and writing data. Usually, the data reading and writing steps increase the processing time of the data processing task. Therefore, when scheduling data processing tasks, the data reading and writing steps are usually separated from the data processing steps.

[0004] In related technologies, scripting languages ​​are typically used to write data processing tasks, such as Python, Java, and Shell. Usually, one script file corresponds to one data processing task. Since big data tasks often require reading multiple files, multiple script files are usually used. For example, data processing task A needs to read files 1, 2, and 3, while data processing task B needs to read files 1 and 4. Therefore, both data processing tasks A and B need to use file 1, requiring the addition of file 1 reading steps in the script files. Similarly, since both data processing tasks A and B use file 1, the steps for reading file 1 are repeated in both tasks, increasing the processing time. Therefore, existing data processing task scheduling methods suffer from low efficiency and low accuracy.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method, apparatus, electronic device, and storage medium for scheduling data processing tasks, in order to at least solve the technical problems of large-scale ETL task management and low efficiency of manually developing scheduling tasks in related technologies.

[0007] According to one aspect of the present invention, a method for scheduling data processing tasks is provided, comprising: obtaining a data processing mapping, wherein the data processing mapping refers to a mapping configuration file obtained by configuring a preset mapping template based on data processing requirements; generating a data processing script based on the data processing mapping, and generating N data processing tasks based on the data processing script, wherein N is a positive integer; constructing a scheduling flowchart based on the N data processing tasks, wherein the scheduling flowchart includes N task nodes, each task node corresponding to a data processing task, and the scheduling flowchart refers to a directed acyclic graph used to record the execution order and dependencies of the data processing tasks; and uploading the N data processing tasks and the scheduling flowchart to a batch scheduling system, wherein the batch scheduling system performs task scheduling.

[0008] Furthermore, the data processing mapping is obtained through the following steps: receiving data processing requirements, wherein the data processing requirements include the following information: business scenario, data source, target table structure, and data transformation logic; filling in the preset mapping template according to the template mapping description and the data processing requirements to obtain the data processing mapping, wherein the data processing mapping includes the following mapping information: source table information, target table information, data processing logic information, field mapping information, and task dependency information.

[0009] Further, the step of generating a data processing script based on the data processing mapping, and generating N data processing tasks based on the data processing script, includes: reading the target table information in the data processing mapping, and generating a table creation statement based on the target table information, wherein the table creation statement is used to create a target table in the target database; reading the data processing logic information in the data processing mapping, and generating a data processing logic statement based on the data processing logic information and the table creation statement; and encapsulating the table creation statement and the data processing logic statement to obtain N data processing tasks.

[0010] Furthermore, the step of constructing a scheduling flowchart based on N data processing tasks includes: obtaining preset task configuration information, wherein the preset task configuration information includes: pre-task attribute information and dependent task attribute information; performing data flow analysis based on the preset task configuration information and the data processing mapping to obtain data flow analysis results, wherein the data flow analysis results are used to record the input-output data relationship of the N data processing tasks during batch task execution; and constructing the scheduling flowchart based on the data flow analysis results.

[0011] Further, the step of constructing the scheduling flowchart based on the data flow analysis results includes: creating an initial flowchart, wherein the initial flowchart includes at least a start node and an end node; creating N task nodes in the initial flowchart based on the N data processing tasks; creating directed edges between the N task nodes in the initial flowchart based on the input-output data relationships recorded in the data flow analysis results, wherein each directed edge connects two task nodes, and the direction of the directed edge indicates the data flow direction; and obtaining the scheduling flowchart when all the input-output data relationships recorded in the data flow analysis results are converted into the directed edges.

[0012] Furthermore, before uploading the N data processing tasks and the scheduling flowchart to the batch scheduling system, the method further includes: detecting the scheduling flowchart and the task nodes based on the batch task scheduling principle, and obtaining a detection result; when the detection result indicates that the scheduling flowchart and all the task nodes have passed the detection, uploading the N data processing tasks and the scheduling flowchart to the batch scheduling system.

[0013] Furthermore, the step of detecting the scheduling flowchart and the task nodes based on the batch task scheduling principle to obtain detection results includes: checking the cyclic dependency of each task node in the scheduling flowchart to obtain a first detection result; determining that the task node passes the detection if the first detection result indicates that the task node does not have a cyclic dependency; checking the connectivity of the scheduling flowchart to obtain a second detection result; and determining that the scheduling flowchart passes the detection if the second detection result indicates that the topology of the scheduling flowchart is completely unobstructed.

[0014] According to another aspect of the present invention, a scheduling method apparatus for data processing tasks is also provided, comprising: an acquisition unit, configured to acquire a data processing mapping, wherein the data processing mapping refers to a mapping configuration file obtained by configuring a preset mapping template based on data processing requirements; a generation unit, configured to generate a data processing script based on the data processing mapping, and generate N data processing tasks based on the data processing script, wherein N is a positive integer; a construction unit, configured to construct a scheduling flowchart based on the N data processing tasks, wherein the scheduling flowchart includes N task nodes, each task node corresponding to one data processing task, and the scheduling flowchart refers to a directed acyclic graph used to record the execution order and dependencies of the data processing tasks; and an upload unit, configured to upload the N data processing tasks and the scheduling flowchart to a batch scheduling system, wherein the batch scheduling system performs task scheduling.

[0015] Furthermore, the data processing task scheduling method apparatus further includes: a receiving module for receiving data processing requirements, wherein the data processing requirements include the following information: business scenario, data source, target table structure, and data transformation logic; and a filling module for filling in a preset mapping template according to the template mapping description and the data processing requirements to obtain the data processing mapping, wherein the data processing mapping includes the following mapping information: source table information, target table information, data processing logic information, field mapping information, and task dependency information.

[0016] Further, the generation unit includes: a first generation module, used to read the target table information in the data processing mapping and generate a table creation statement based on the target table information, wherein the table creation statement is used to create a target table in the target database; a second generation module, used to read the data processing logic information in the data processing mapping and generate a data processing logic statement based on the data processing logic information and the table creation statement; and an encapsulation module, used to encapsulate the table creation statement and the data processing logic statement to obtain N data processing tasks.

[0017] Further, the construction unit includes: an acquisition module for acquiring preset task configuration information, wherein the preset task configuration information includes: pre-task attribute information and dependent task attribute information; an analysis module for performing data flow analysis based on the preset task configuration information and the data processing mapping to obtain data flow analysis results, wherein the data flow analysis results are used to record the input-output data relationship of N data processing tasks during batch task execution; and a construction module for constructing the scheduling flowchart based on the data flow analysis results.

[0018] Further, the construction module includes: a first creation submodule for creating an initial flowchart, wherein the initial flowchart includes at least a start node and an end node; a second creation submodule for creating N task nodes in the initial flowchart based on N data processing tasks; a third creation submodule for creating directed edges between the N task nodes in the initial flowchart based on the input-output data relationships recorded in the data flow analysis results, wherein each directed edge connects two task nodes, and the direction of the directed edge indicates the data flow direction; and a first determination submodule for obtaining the scheduling flowchart when all the input-output data relationships recorded in the data flow analysis results have been converted into the directed edges.

[0019] Furthermore, the data processing task scheduling device further includes: a detection module, used to detect the scheduling flowchart and the task nodes based on the batch task scheduling principle, and obtain a detection result; and an upload module, used to upload N data processing tasks and the scheduling flowchart to the batch scheduling system when the detection result indicates that the scheduling flowchart and all the task nodes have passed the detection.

[0020] Furthermore, the detection module includes: a first checking submodule, used to check the cyclic dependency of each task node in the scheduling flowchart to obtain a first checking result; a second determining submodule, used to determine that the task node passes the detection if the first checking result indicates that the task node does not have a cyclic dependency; a second checking submodule, used to check the connectivity of the scheduling flowchart to obtain a second checking result; and a third determining submodule, used to determine that the scheduling flowchart passes the detection if the second checking result indicates that the topology of the scheduling flowchart is completely unobstructed.

[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a scheduling method for the data processing task described in any one of the above embodiments.

[0022] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing task scheduling method described in any one of the above embodiments.

[0023] This invention proposes a method for scheduling data processing tasks. First, a data processing mapping is obtained, which is a mapping configuration file obtained by configuring a preset mapping template based on data processing requirements. Then, a data processing script is generated based on the data processing mapping, and N data processing tasks are generated based on the data processing script, where N is a positive integer. Next, a scheduling flowchart is constructed based on the N data processing tasks, where the scheduling flowchart includes N task nodes, each corresponding to a data processing task. The scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of the data processing tasks. Finally, the N data processing tasks and the scheduling flowchart are uploaded to a batch scheduling system for task scheduling.

[0024] This invention first automates the transformation of business requirements into system scripts through data processing mapping configuration, avoiding the tediousness of manual coding. Then, it automatically generates N data processing tasks and constructs a directed acyclic graph (DAG), clearly depicting the dependencies and execution order between tasks, eliminating the problems of circular dependencies and ensuring the orderly lineage of data. Finally, the tasks and DAG are uploaded to a batch scheduling system for intelligent scheduling. Even when faced with large-scale tasks, it can automatically identify and efficiently execute upstream and downstream tasks, greatly improving development efficiency and the level of automation in data processing. This thoroughly optimizes the ETL process of commercial bank data warehouses, thereby solving the technical problems of large-scale ETL task management and low efficiency of manually developing and scheduling tasks in related technologies. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0026] Figure 1 This is a flowchart of an optional data processing task scheduling method according to an embodiment of the present invention;

[0027] Figure 2 This is a functional block diagram of an optional commercial financial institution ETL task automatic generation and scheduling system according to an embodiment of the present invention.

[0028] Figure 3 This is a flowchart of an optional method for automatically generating and scheduling ETL tasks for commercial financial institutions according to an embodiment of the present invention;

[0029] Figure 4 This is a flowchart of an optional method for automatically generating ETL task workflows based on Airflow according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of an optional data processing task scheduling device according to an embodiment of the present invention;

[0031] Figure 6 This is a hardware structure block diagram of an electronic device (or mobile device) for a scheduling method for data processing tasks according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0035] ETL, or Extract-Transform-Load, refers to the process of extracting, transforming, and loading data.

[0036] The Data Source Layer (ODS) is the raw data layer, storing the most original data obtained from the business system. The data and data structure are consistent with the data source, and it is the foundation layer for all upper layers of the data warehouse.

[0037] The Integrated Model Layer (IML) is theme-oriented. Based on the main business operations of the business systems, focusing on the data structure of the source systems, and incorporating industry implementation experience, it defines subject domains. As the research on business systems deepens, table and field-level analysis is conducted, the data warehouse data dictionary is maintained, and data from across the bank is continuously integrated. Data from various source systems is cleaned and transformed, and this layer's data model is updated / iterated. By abstracting relatively independent themes, the impact of changes in the ODS layer on the data warehouse is shielded and absorbed, ensuring the relative stability of the upper-layer model.

[0038] The Common Processing Layer (ICL) takes a business perspective and, based on the needs of existing business reports, marts, and data applications, extracts common requirements and performs pre-association, pre-aggregation, and pre-processing of data to reduce data redundancy and lower the complexity of application processing and data use. This results in the construction of a public data platform that supports applications and provides shared data access services.

[0039] DAG, Directed Acyclic Graph, in this invention, ETL tasks, their predecessor tasks, and the relationship between ETL tasks and their predecessor tasks are written into a DAG instance and represented in the graph in the form of nodes and directed edges.

[0040] It should be noted that the data processing task scheduling method and apparatus of the present invention can be used in the field of big data processing technology for batch management and scheduling of ETL tasks, and can also be used in any field other than big data processing for batch management and scheduling of ETL tasks. The present invention does not limit the application field of the data processing task scheduling method and apparatus.

[0041] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, processing, transmission, provision, disclosure, use, and handling of such data must comply with the laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and it does not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0042] The information collection (e.g., user voice, video, and text collection) and analysis operations involved in this invention have provided users with corresponding operation entry points during execution, allowing users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process enters the expert decision-making process.

[0043] The following embodiments of the present invention can be applied to various systems / applications / devices that require ETL task generation and batch management scheduling, enabling automatic construction and intelligent scheduling of ETL tasks based on directed acyclic graphs (DAGs) of data processing mappings. By encapsulating data processing logic and task scheduling rules within the data processing mapping, the present invention automates the conversion from mapping files to task scripts and then to scheduling flowcharts, significantly improving the development efficiency of ETL tasks, reducing errors and redundant code, and solving the problem of large-scale task management through automatic DAG construction, thus achieving efficient, intelligent, and automated large-scale task scheduling.

[0044] The present invention will now be described in detail with reference to various embodiments.

[0045] Example 1

[0046] According to an embodiment of the present invention, an embodiment of a scheduling method for data processing tasks is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] The implementation subject of this invention is the data warehouse system of a commercial bank and its related ETL (Extract-Transform-Load) automation tools and batch scheduling systems (such as Airflow). It relies on the efficient operation of automation tools and scheduling systems to reduce manual intervention, thereby improving the generation and scheduling efficiency of ETL tasks.

[0048] During implementation, data engineers or ETL developers first configure a data processing mapping based on business needs and the data model. This configuration file contains the key attributes of the ETL tasks and the data processing logic. Subsequently, the implementation system automatically generates data processing scripts based on the mapping and further generates data processing tasks. The number of tasks is determined by the actual needs of the data warehouse. Next, the implementation system automatically constructs a directed acyclic graph (DAG) representing task dependencies and execution order, enabling visualization and logical management of the task flow. Finally, the generated tasks and DAG are uploaded to a batch scheduling system, which is responsible for task scheduling, ensuring that each task is executed correctly and in a timely manner according to its position and dependencies in the DAG.

[0049] The embodiments of the present invention will be described in detail below with reference to specific implementation steps.

[0050] Figure 1 This is a flowchart of an optional data processing task scheduling method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0051] Step S101: Obtain the data processing mapping, where the data processing mapping refers to the mapping configuration file obtained after configuring the preset mapping template based on the data processing requirements.

[0052] Specifically, the data processing mapping is a configuration file obtained by configuring a preset mapping template based on business requirements. It contains key information and rules required to execute ETL tasks, such as data source, target table structure, data transformation logic, and how to map this information to specific database operations. The preset mapping template is used to provide a standardized framework and fields to quickly and accurately fill in the above requirements. The structure and field design of the template are based on the experience and best practices of previous ETL task implementations.

[0053] Data processing requirements include business scenarios (such as customer behavior analysis, credit assessment, etc.), data sources (such as database tables, log files, etc.), target table structure (defining the data table structure output by the ETL task), and data transformation logic (specifying how to extract, transform, and load data from the source data into the target table).

[0054] Optionally, the data processing mapping is obtained through the following steps: receiving data processing requirements, which include the following information: business scenario, data source, target table structure, and data transformation logic; filling in the preset mapping template according to the template mapping description and data processing requirements to obtain the data processing mapping, which includes the following mapping information: source table information, target table information, data processing logic information, field mapping information, and task dependency information.

[0055] Specifically, source table information refers to the detailed information of the original data tables involved in the data extraction (Extract) phase of the ETL process, including: the name of the source data table, the name of the database it resides in, the structure of the data table (field names, field types), the data storage format, and the data update frequency. Source table information is fundamental to the ETL task, determining the data extraction method and the construction of subsequent processing logic.

[0056] Target table information refers to the detailed information of the target data table involved in the data loading phase of the ETL process. This includes the target data table name, the name of the database it resides in, the table structure (field names, field types), data storage format, data update strategy (such as append, overwrite, replace_into), and possible data partitioning strategies. Target table information determines the final storage method and format of the data, and is the data presentation form after the ETL task is completed.

[0057] Data processing logic information refers to a detailed description of the rules and algorithms used to process data during the data transformation phase, including data cleaning logic, field calculation logic, data aggregation logic, and data format conversion logic. Data processing logic information is the core of an ETL task, determining how source data is transformed into target data, and is a crucial step in ensuring data quality and meeting business requirements.

[0058] Field mapping information refers to the correspondence and conversion rules between fields in the source table and fields in the target table. This includes not only direct mapping of field names but also potentially field type conversions, field value mappings, and field-level data processing rules such as field splitting, merging, and formatting. Field mapping information ensures the correctness and consistency of each field during the transformation from source data to target data, guaranteeing the accuracy of ETL task execution.

[0059] Task dependency information refers to the upstream and downstream relationships between ETL tasks and the data processing process. It describes which tasks are prerequisites for the current task and which tasks depend on the output of the current task, thus ensuring the continuity of the data processing flow and the integrity of the data. Task dependency information is the foundation for building a directed acyclic graph (DAG), which can guide batch scheduling systems (such as Airflow) to intelligently schedule the execution order of tasks, avoid circular dependencies, and ensure the correct loading and processing of data.

[0060] By combining the preset template with the automated filling mechanism in step S101, the workload of task configuration can be greatly reduced, the deployment speed of ETL tasks can be accelerated, and the obtained standardized data processing mapping can ensure the consistency and accuracy of the data processing process, reduce human error and improve data quality. In addition, the task dependency information contained in the mapping helps to automatically build a directed acyclic graph (DAG), ensure the correct execution order between tasks, avoid circular dependencies, and enhance the intelligence of scheduling.

[0061] Step S102: Generate a data processing script based on the data processing mapping, and generate N data processing tasks based on the data processing script, where N is a positive integer.

[0062] Specifically, data processing scripts are the concrete instructions for executing ETL tasks. They act directly on the database and serve as the execution vehicle for data transformation and loading. They contain a series of database operation instructions or programming scripts, primarily SQL scripts, used to describe how to extract data from source tables, clean and transform the data, and load the processed data into target tables. Data processing scripts are automatically generated based on the data processing logic information contained in the mapping, ensuring the consistency and accuracy of task execution.

[0063] A data processing task is encapsulated by a data processing script and represents an independent processing unit in the ETL process. A data processing task may consist of one or more data processing scripts, depending on the complexity of the data processing and the boundaries of the transaction. In this invention, the data processing task is the smallest execution unit in the ETL process; it can be executed independently or scheduled for execution according to dependencies in a batch scheduling system. The generation and scheduling of data processing tasks are based on the task dependency information in the data processing mapping, ensuring the continuity of the data processing flow and the integrity of the data.

[0064] Optionally, step S102 includes: reading the target table information in the data processing mapping and generating a table creation statement based on the target table information, wherein the table creation statement is used to create a target table in the target database; reading the data processing logic information in the data processing mapping and generating a data processing logic statement based on the data processing logic information and the table creation statement; and encapsulating the table creation statement and the data processing logic statement to obtain N data processing tasks.

[0065] In this embodiment of the invention, the definition of the target data table in the mapping file is parsed, including the table structure, field information, data type, storage location, etc., which can provide a basis for generating the table creation statement. Based on the target table information, the DDL (Data Definition Language) part of the database query language (such as SQL) is constructed, namely the CREATE TABLE statement, which is used to create the target data table that meets the requirements in the target database.

[0066] In addition, parsing the rules about data transformation and processing in the mapping file, including field calculations, data cleaning, aggregation operations, and formatting, can provide guidance for generating data processing logic statements. Based on the data processing logic information, the DML (Data Manipulation Language) part of the database query language (such as SQL) can be constructed, namely SELECT, INSERT, UPDATE, DELETE, etc., which are used to extract, transform, and load data.

[0067] The generated statements are encapsulated into independent task units. Each task unit may correspond to one or more database operations, forming N data processing tasks. These tasks can be SQL scripts, Shell scripts, or other forms of executable code. The creation and execution order of all tasks follows the task dependency information in the mapping file.

[0068] For example, when processing transaction data, the data processing mapping might include the following information: the target table is the transaction details table, with fields including transaction date, transaction amount, and transaction type; the data processing logic information includes extracting transaction data from the source table, standardizing the date format, converting the transaction amount to currency, and loading the data into the target table. The table creation statement generated by reading the target table information is used, such as: `CREATE TABLE transaction_details(transaction_date DATE, transaction_amount DECIMAL, transaction_type VARCHAR, ...)`; then, the data processing logic information is read to construct the data processing logic statement, such as `INSERT INTO transaction_details SELECTDATE_FORMAT(trans_date,'%Y-%m-%d'), trans_amount*100, trans_type FROM source_transactions`. Finally, the above script is encapsulated into a data processing task and scheduled according to task dependencies to ensure data accuracy and timeliness.

[0069] For example, when updating customer credit information, the data processing mapping might include a source table containing credit application records and basic customer information, a target table containing credit customer profiles, and data processing logic information including data cleaning (such as removing duplicate records), field calculations (such as credit scoring), and data integration (multi-table JOIN operations). The system generates table creation statements based on this information and simultaneously constructs data processing logic statements, which may involve complex SQL join queries and data calculations, such as: `SELECT c.first_name, c.last_name, l.loan_amount, l.interest_rate, calculate_credit_score(l) AS credit_score FROM customers c JOIN loans l ON c.customer_id=l.customer_id`. The encapsulated data processing tasks are automatically scheduled according to the dependencies of the customer profile update process, ensuring the real-time nature and accuracy of credit information while reducing the complexity of ETL development and maintenance.

[0070] The above step S102 automatically generates table creation and data processing scripts by reading the mapping file, which reduces the workload of manual coding and improves efficiency. The flexible configuration of data processing logic information supports the generation of customized data processing scripts according to different business needs and data source characteristics, improving the flexibility and accuracy of data processing. Encapsulating the data processing scripts into independent tasks helps to standardize and modularize task management, facilitates subsequent scheduling and monitoring, and also supports task reuse and optimization. Finally, through the task dependency information in the mapping file, the automatically generated data processing tasks can automatically identify and comply with the dependencies between tasks, ensuring the correct processing and loading order of data.

[0071] Step S103: Construct a scheduling flowchart based on N data processing tasks. The scheduling flowchart includes N task nodes, each task node corresponding to a data processing task. The scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of data processing tasks.

[0072] It's important to note that constructing a scheduling flowchart is the process of visually representing N data processing tasks in the form of a directed acyclic graph (DAG) according to their interdependencies and execution order. This step is crucial for ensuring the correct execution of the data processing flow, enabling the batch scheduling system to intelligently schedule tasks, avoid circular dependencies, and ensure the continuity of data flow and the orderliness of tasks.

[0073] Specifically, a Directed Acyclic Graph (DAG) is a special graph structure where edges are directed and never form cyclic paths. In this invention, DAGs are used to describe and manage the sequential execution relationships between ETL tasks, ensuring the correctness and efficiency of the data processing flow. The use of DAGs enables batch scheduling systems (such as Airflow) to intelligently determine the execution order of tasks, ensuring that all tasks are executed correctly according to their dependencies, avoiding circular dependencies—that is, no situation where task A depends on task B, and task B also depends on task A.

[0074] Optionally, step S103 includes: obtaining preset task configuration information, wherein the preset task configuration information includes: pre-task attribute information and dependent task attribute information; performing data flow analysis based on the preset task configuration information and data processing mapping to obtain data flow analysis results, wherein the data flow analysis results are used to record the input and output data relationships of N data processing tasks during batch task execution; and constructing a scheduling flowchart based on the data flow analysis results.

[0075] Specifically, the preset task configuration information defines the dependencies between data processing tasks, including which tasks are prerequisites for the current task and which tasks depend on the output of the current task. This information forms the basis for further data flow analysis and the construction of the scheduling flowchart. Prerequisite task attribute information refers to the information about tasks that must be completed before the data processing task is executed, including the name and execution status of the prerequisite task. In the ETL process, prerequisite tasks typically refer to tasks in the data extraction phase or other tasks that depend on the current task. Dependency task attribute information refers to the information about subsequent tasks that need to be executed after the current data processing task is executed. In the ETL process, dependent tasks are often tasks in the data loading phase or other tasks that perform further processing based on the output of the current task.

[0076] In this embodiment of the invention, the pre-task attribute information ensures that all necessary data and conditions are ready before the data processing task is executed, which helps to avoid duplication and confusion in data processing, ensures the continuity of the data processing flow and the integrity of the data. The task attribute information ensures that the results of the task output can be correctly used by subsequent tasks, supports the continuity of the data processing flow, and helps the scheduling system to intelligently determine the priority of task execution, thereby improving the overall processing efficiency.

[0077] Optionally, the step of constructing a scheduling flowchart based on the data flow analysis results includes: creating an initial flowchart, wherein the initial flowchart includes at least a start node and an end node; creating N task nodes in the initial flowchart based on N data processing tasks; creating directed edges between the N task nodes in the initial flowchart based on the input-output data relationships recorded in the data flow analysis results, wherein each directed edge connects two task nodes, and the direction of the directed edge indicates the data flow direction; and obtaining the scheduling flowchart when all input-output data relationships recorded in the data flow analysis results have been transformed into directed edges.

[0078] It should be noted that creating an initial flowchart with a start node and an end node at the beginning of the scheduling flowchart provides a framework for adding data processing task nodes and directed edges later. Each task node represents an independent data processing task, including task attributes and data processing logic. Each directed edge represents the direction of data flow from one task to another. Establishing directed edges ensures the sequentiality and coherence of data processing, that is, the data processing flow follows the direction of data flow, from the source task to the target task, without reverse loops, thus avoiding logical confusion in data processing.

[0079] In this embodiment of the invention, task dependencies are visualized through a scheduling flowchart, which graphically presents the dependencies between tasks, making the management and monitoring of the ETL process more intuitive and easier to understand. Intelligent scheduling is achieved through a Directed Acyclic Graph (DAG), enabling the batch scheduling system to intelligently identify and schedule tasks, ensuring the efficiency and accuracy of data processing. Simultaneously, the scheduling flowchart ensures the maintainability of the data processing flow; task dependencies can be updated simply by adjusting the mapping and configuration information, without requiring manual code modification. Furthermore, the scheduling flowchart, built based on a directed acyclic graph, ensures the continuity and acyclicity of task execution, avoiding data processing deadlocks or anomalies caused by circular dependencies.

[0080] Step S104: Upload the N data processing tasks and scheduling flowcharts to the batch scheduling system, and the batch scheduling system will execute the task scheduling.

[0081] Specifically, batch scheduling systems (such as Airflow) refer to workflow management platforms used to automate the management and monitoring of data processing flows, especially ETL (Extract, Transform, Load) tasks, using directed acyclic graphs (DAGs) to describe and execute the flow between tasks.

[0082] Optionally, before performing step S104, the method further includes: detecting the scheduling flowchart and task nodes based on the batch task scheduling principle, and obtaining the detection result; when the detection result indicates that the scheduling flowchart and all task nodes have passed the detection, uploading the N data processing tasks and the scheduling flowchart to the batch scheduling system.

[0083] It should be noted that during the automatic generation and scheduling of ETL tasks, ensuring the correctness of the scheduling flowchart and the health status of the task nodes is crucial. This involves checking the scheduling flowchart and task nodes to confirm that they comply with the batch task scheduling principles. After passing the check, they are uploaded to the batch scheduling system for execution.

[0084] Specifically, batch task scheduling principles can include the principle of no circular dependencies and the principle of data flow continuity. The principle of no circular dependencies means that the dependencies between tasks in the DAG structure of batch scheduling must be acyclic; that is, there should be no situation where task A depends on task B, and task B also depends on task A. This ensures the correct execution order of tasks and avoids scheduling deadlocks or infinite loops. The principle of data flow continuity means that the data flow between data processing tasks must be continuous and coherent; that is, the output of each task should become the input of the next task, ensuring the integrity and consistency of data processing.

[0085] Optionally, the step of detecting the scheduling flowchart and task nodes based on the batch task scheduling principle to obtain the detection result includes: checking the circular dependency of each task node in the scheduling flowchart to obtain a first detection result; if the first detection result indicates that the task node does not have a circular dependency, determining that the task node passes the detection; checking the connectivity of the scheduling flowchart to obtain a second detection result; if the second detection result indicates that the topology of the scheduling flowchart is completely unobstructed, determining that the scheduling flowchart passes the detection.

[0086] Furthermore, the first check result refers to the result obtained by checking for circular dependencies in task nodes based on the principle of no circular dependencies. The purpose is to detect whether circular dependencies exist in the scheduling flowchart, meaning there is no situation where task node A depends on task node B, and B also depends on A. Circular dependencies can cause tasks to fail to execute correctly, and may even lead to deadlock in the scheduling system. By checking the directed acyclic graph structure of the scheduling flowchart, it can be ensured that the relationships between the predecessor and dependent tasks of each task node are correct and that no closed loops are formed. When the first check result indicates that a task node does not exhibit circular dependencies, it can be considered that the task node has passed the check and can be safely uploaded to the batch scheduling system.

[0087] The second check result refers to the connectivity check obtained by performing a connectivity check on the task nodes based on the data flow continuity principle. The purpose is to ensure that the topology of the scheduling flowchart is connected, meaning that all task nodes form a coherent data processing link, with no isolated task nodes. Connectivity is a crucial condition for ensuring the integrity of the data processing flow. Only when the entire topology of the scheduling flowchart is flowing smoothly, i.e., all task nodes can be scheduled and executed according to the direction of data flow, can the scheduling flowchart be considered to have passed the check.

[0088] Through the above steps S101 to S104, a data processing mapping can be obtained first. The data processing mapping refers to the mapping configuration file obtained after configuring a preset mapping template based on data processing requirements. Then, a data processing script is generated based on the data processing mapping, and N data processing tasks are generated based on the data processing script, where N is a positive integer. Then, a scheduling flowchart is constructed based on the N data processing tasks. The scheduling flowchart includes N task nodes, each task node corresponding to a data processing task. The scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of data processing tasks. Finally, the N data processing tasks and the scheduling flowchart are uploaded to the batch scheduling system, and the batch scheduling system executes the task scheduling.

[0089] This invention first automates the conversion of business requirements into system scripts through data processing mapping configuration, avoiding the tediousness of manual coding. Then, it automatically generates N data processing tasks and constructs a directed acyclic graph (DAG), clearly depicting the dependencies and execution order between tasks, eliminating the problem of circular dependencies, and ensuring the orderly lineage of data. Finally, the tasks and DAG are uploaded to a batch scheduling system for intelligent scheduling. Even when faced with large-scale tasks, it can automatically identify and efficiently execute upstream and downstream tasks, greatly improving development efficiency and the level of automation in data processing. This thoroughly optimizes the ETL process of commercial bank data warehouses, thereby solving the technical problems of large-scale ETL task management and low efficiency of manually developing and scheduling tasks in related technologies.

[0090] The present invention will now be described in conjunction with another specific embodiment.

[0091] Commercial financial institutions generate massive amounts of data across multiple business systems in their daily operations. As these institutions grow, they need to process this data, which spans multiple systems, more intelligently and eliminate data silos. Building a data warehouse requires executing numerous ETL tasks, such as ETL tasks at the data source layer (ODS), data integration layer (IML), common processing layer (ICL), and application mart layer, with their dependencies progressively increasing.

[0092] During the implementation of a data warehouse, data marts from different business departments are built according to data layers, and there are also cross-departmental data dependencies. Often, department A depends on data from department B, and department B also depends on data from department A. Therefore, as the number of tasks increases during the data warehouse construction process, the relationships between ETL tasks become more intricate and the data lineage becomes chaotic, making it difficult to attribute problems. Furthermore, circular dependencies between tasks can lead to serious problems such as data scheduling errors in the entire data warehouse and a huge impact on business.

[0093] In existing technical solutions, manual coding is usually adopted. Custom code development is carried out for each data table according to the data model. The steps of building ETL tasks are very complicated. It is also necessary to manually configure the dependencies between upstream and downstream ETL tasks. Furthermore, due to the large scale of task management, the design method of processing a single data table model for a single task results in a huge task management scale. In summary, the manual development method is inefficient, cannot guarantee code quality, and has high testing costs.

[0094] Based on the above, this invention provides a method for automatically generating and scheduling ETL tasks for commercial financial institutions. By configuring and parsing unified template mapping information, it automatically generates table creation statements, ETL logic SQL, and ETL tasks, automatically derives batch task dependencies, and constructs a directed acyclic graph (DAG) for batch task execution, thereby solving the technical problems of low efficiency and high testing costs in manual development.

[0095] Figure 2 This is a functional block diagram of an optional commercial financial institution ETL task automatic generation and scheduling system according to an embodiment of the present invention, such as... Figure 2 As shown, the system includes: an ETL task configuration and parsing module, an ETL task automatic generation module, a batch task dependency detection module, an ETL task process automatic generation module, and an ETL task operation monitoring and management module.

[0096] Based on the above system, the following can be performed: Figure 3 Implementation steps Figure 3 This is a flowchart of an optional method for automatically generating and scheduling ETL tasks for commercial financial institutions according to an embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps:

[0097] S1. Fill in the template mapping according to the template instructions.

[0098] S2, parse the template mapping to obtain the mapping configuration information.

[0099] S3 automatically generates result table information and ETL execution SQL commands based on mapping configuration information.

[0100] S4 automatically generates ETL task configuration information based on the result table information and SQL commands, and inputs the ETL task configuration information into the batch scheduling system to trigger batch ETL tasks.

[0101] The templates provided in the embodiments of the present invention are illustrated in Table 1 below:

[0102] Table 1

[0103]

[0104]

[0105] Regarding the available parameters ${start_dt}' and (${start_dt} in the table, during the ETL task execution, a corresponding date will be generated based on the run_date (task execution date) of the ETL task information. If the run_date is 20230101, then ${start_dt} and ${end_dt} will be replaced with 2023-01-01 in the program. If the run_date is between 20230101 and 20230201, then ${start_dt} = 2023-01-01 and ${end_dt} = 2023-02-01. Based on these characteristics, the run_date of the ETL task information can be modified to re-run historical data.

[0106] The embodiments of this invention use Airflow as the batch scheduling system, which is a task management system based on a directed acyclic graph (DAG). It can automatically deduce the upstream and downstream task relationships, i.e. the dependency relationships, based on the directed acyclic graph (DAG), automatically construct the directed acyclic graph of the batch task execution process, and automatically generate task instructions based on each task node of the directed acyclic graph, so as to realize the complete automation and intelligence of the entire process from generation to execution of the directed acyclic graph.

[0107] Figure 4 This is a flowchart of an optional method for automatically generating ETL task workflows based on Airflow according to an embodiment of the present invention, such as... Figure 4 As shown, the method includes the following steps:

[0108] T1 automatically starts the batch task process at regular intervals, initializes the start task, and generates the start node (start) of the directed acyclic graph.

[0109] Alternatively, the scheduled task can be set to run once a day (i.e. every 24 hours). Then, the ETL model configuration information can be read based on the business line ID and group ID to determine whether the task for the day has been generated. If it has not been generated, the ETL task is generated and inserted into the ETL task run record table. If the task for the day has been generated, it is ignored. This method is mainly used for the rerun mechanism.

[0110] T2 reads the ETL task configuration information, iterates through the batch dependency check configuration information, generates a batch detection task, sets the upstream task of the batch detection task as the starting task, and sets the detection SQL command to the detection command executed by the batch detection task.

[0111] T3 reads the preset ETL configuration information, deduces and builds upstream dependencies based on the configuration information of the ETL source table and target table, deduces and generates downstream ETL tasks, and automatically generates ETL task SQL execution commands.

[0112] T4 automatically generates the end node, automatically generates upstream and downstream dependencies based on preset configuration information, and automatically constructs a directed acyclic graph for batch task execution. It sequentially builds a directed acyclic graph (DAG) based on the computation tasks in the task script. This directed acyclic graph is used to guide the execution flow of ETL tasks.

[0113] T5 writes the results of the entire DAG execution to the success file, marking the completion of the batch task for this business line.

[0114] The embodiments of the present invention automatically generate table creation statements, ETL logic SQL, and ETL tasks by developing, managing, and parsing unified template mapping information. It can also automatically deduce batch task dependencies and construct a directed acyclic graph (DAG) for batch task execution based on the generated ETL task information, thus solving the technical problems of low efficiency and high testing costs of manual development methods.

[0115] The invention will now be described in conjunction with another alternative embodiment.

[0116] Example 2

[0117] The data processing task scheduling device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0118] Figure 5 This is a schematic diagram of an optional data processing task scheduling device according to an embodiment of the present invention, such as... Figure 5 As shown, the device may include: an acquisition unit 51, a generation unit 52, a construction unit 53, and an upload unit 54.

[0119] The acquisition unit 51 is used to acquire the data processing mapping, which refers to the mapping configuration file obtained after configuring the preset mapping template based on the data processing requirements.

[0120] The generation unit 52 is used to generate a data processing script based on the data processing mapping, and to generate N data processing tasks based on the data processing script, where N is a positive integer.

[0121] Building unit 53 is used to construct a scheduling flowchart based on N data processing tasks. The scheduling flowchart includes N task nodes, each task node corresponds to a data processing task, and the scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of data processing tasks.

[0122] Upload unit 54 is used to upload N data processing tasks and scheduling flowcharts to the batch scheduling system, which then performs task scheduling.

[0123] The scheduling device for the aforementioned data processing tasks can first obtain the data processing mapping through the acquisition unit 51. The data processing mapping refers to the mapping configuration file obtained after configuring a preset mapping template based on the data processing requirements. Then, the generation unit 52 generates a data processing script based on the data processing mapping and generates N data processing tasks based on the data processing script, where N is a positive integer. Then, the construction unit 53 constructs a scheduling flowchart based on the N data processing tasks. The scheduling flowchart includes N task nodes, each task node corresponding to a data processing task. The scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of the data processing tasks. Finally, the upload unit 54 uploads the N data processing tasks and the scheduling flowchart to the batch scheduling system, which then performs task scheduling.

[0124] In this embodiment of the invention, business requirements are first automatically converted into system scripts through data processing mapping configuration, avoiding the tediousness of manual coding. Then, the automatically generated N data processing tasks and the constructed Directed Acyclic Graph (DAG) clearly depict the dependencies and execution order between tasks, eliminating the trouble of circular dependencies and ensuring the orderly data lineage. Finally, the tasks and DAG are uploaded to the batch scheduling system to achieve intelligent scheduling. Even when faced with large-scale tasks, it can automatically identify and efficiently execute upstream and downstream tasks, greatly improving development efficiency and the level of automation in data processing. This thoroughly optimizes the ETL process of commercial bank data warehouses, thereby solving the technical problems of large-scale ETL task management and low efficiency of manually developing and scheduling tasks in related technologies.

[0125] Optionally, the data processing task scheduling method apparatus further includes: a receiving module for receiving data processing requirements, wherein the data processing requirements include the following information: business scenario, data source, target table structure, and data transformation logic; and a filling module for filling in a preset mapping template according to the template mapping description and data processing requirements to obtain a data processing mapping, wherein the data processing mapping includes the following mapping information: source table information, target table information, data processing logic information, field mapping information, and task dependency information.

[0126] Optionally, the generation unit includes: a first generation module, used to read target table information from the data processing mapping and generate table creation statements based on the target table information, wherein the table creation statements are used to create target tables in the target database; a second generation module, used to read data processing logic information from the data processing mapping and generate data processing logic statements based on the data processing logic information and the table creation statements; and an encapsulation module, used to encapsulate the table creation statements and data processing logic statements to obtain N data processing tasks.

[0127] Optionally, the construction unit includes: an acquisition module for acquiring preset task configuration information, wherein the preset task configuration information includes: pre-task attribute information and dependent task attribute information; an analysis module for performing data flow analysis based on the preset task configuration information and data processing mapping to obtain data flow analysis results, wherein the data flow analysis results are used to record the input and output data relationships of N data processing tasks during batch task execution; and a construction module for constructing a scheduling flowchart based on the data flow analysis results.

[0128] Optionally, the construction module includes: a first creation submodule for creating an initial flowchart, wherein the initial flowchart includes at least a start node and an end node; a second creation submodule for creating N task nodes in the initial flowchart based on N data processing tasks; a third creation submodule for creating directed edges between the N task nodes in the initial flowchart based on the input-output data relationships recorded in the data flow analysis results, wherein each directed edge connects two task nodes, and the direction of the directed edge indicates the data flow direction; and a first determination submodule for obtaining a scheduling flowchart when all input-output data relationships recorded in the data flow analysis results have been converted into directed edges.

[0129] Optionally, the data processing task scheduling device further includes: a detection module, used to detect the scheduling flowchart and task nodes based on the batch task scheduling principle and obtain the detection result; and an upload module, used to upload N data processing tasks and the scheduling flowchart to the batch scheduling system when the detection result indicates that the scheduling flowchart and all task nodes have passed the detection.

[0130] Optionally, the detection module includes: a first checking submodule, used to check the circular dependency of each task node in the scheduling flowchart and obtain a first checking result; a second determining submodule, used to determine that the task node passes the detection if the first checking result indicates that there is no circular dependency; a second checking submodule, used to check the connectivity of the scheduling flowchart and obtain a second checking result; and a third determining submodule, used to determine that the scheduling flowchart passes the detection if the second checking result indicates that the topology of the scheduling flowchart is completely unobstructed.

[0131] The aforementioned data processing task scheduling device may also include a processor and a memory. The aforementioned acquisition unit 51, generation unit 52, construction unit 53, upload unit 54, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0132] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, a scheduling flowchart can be constructed based on N data processing tasks. These N data processing tasks and the scheduling flowchart are then uploaded to a batch scheduling system for task scheduling.

[0133] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0134] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: obtaining a data processing mapping, wherein the data processing mapping refers to a mapping configuration file obtained after configuring a preset mapping template based on data processing requirements; generating a data processing script based on the data processing mapping, and generating N data processing tasks based on the data processing script, wherein N is a positive integer; constructing a scheduling flowchart based on the N data processing tasks, wherein the scheduling flowchart includes N task nodes, each task node corresponding to a data processing task, and the scheduling flowchart refers to a directed acyclic graph used to record the execution order and dependencies of the data processing tasks; uploading the N data processing tasks and the scheduling flowchart to a batch scheduling system, and having the batch scheduling system perform task scheduling.

[0135] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein a scheduling method for controlling the device where the computer-readable storage medium is located to perform any of the above-mentioned data processing tasks when the computer program is running.

[0136] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to implement the scheduling method for the data processing task described above.

[0137] Figure 6This is a hardware structure block diagram of an electronic device (or mobile device) according to an embodiment of the present invention for a scheduling method for data processing tasks. Figure 6 As shown, an electronic device may include one or more ( Figure 6 The processor (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) is illustrated using processors 602a, 602b, ..., 602n, and a memory 604 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports in the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0138] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0139] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0144] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for scheduling data processing tasks, characterized in that, include: Obtain the data processing mapping, wherein the data processing mapping refers to the mapping configuration file obtained after configuring a preset mapping template based on data processing requirements; A data processing script is generated based on the data processing mapping, and N data processing tasks are generated based on the data processing script, where N is a positive integer; A scheduling flowchart is constructed based on N data processing tasks. The scheduling flowchart includes N task nodes, each task node corresponds to one data processing task, and the scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of the data processing tasks. The N data processing tasks and the scheduling flowchart are uploaded to the batch scheduling system, which then performs the task scheduling.

2. The scheduling method according to claim 1, characterized in that, The data processing mapping is obtained through the following steps: Receive data processing requests, wherein the data processing requests include the following information: business scenario, data source, target table structure, and data transformation logic; The preset mapping template is filled in according to the template mapping description and the data processing requirements to obtain the data processing mapping. The data processing mapping includes the following mapping information: source table information, target table information, data processing logic information, field mapping information, and task dependency information.

3. The scheduling method according to claim 1, characterized in that, The steps of generating a data processing script based on the data processing mapping, and generating N data processing tasks based on the data processing script, include: Read the target table information in the data processing mapping, and generate a table creation statement based on the target table information, wherein the table creation statement is used to create a target table in the target database; Read the data processing logic information in the data processing mapping, and generate a data processing logic statement based on the data processing logic information and the table creation statement; The table creation statement and the data processing logic statement are encapsulated to obtain N data processing tasks.

4. The scheduling method according to claim 1, characterized in that, The steps to construct a scheduling flowchart based on N data processing tasks include: Obtain preset task configuration information, wherein the preset task configuration information includes: prerequisite task attribute information and dependent task attribute information; Data flow analysis is performed based on the preset task configuration information and the data processing mapping to obtain data flow analysis results. The data flow analysis results are used to record the input and output data relationships of N data processing tasks during the batch task execution process. The scheduling flowchart is constructed based on the data flow analysis results.

5. The scheduling method according to claim 4, characterized in that, The steps for constructing the scheduling flowchart based on the data flow analysis results include: Create an initial flowchart, wherein the initial flowchart includes at least a start node and an end node; Based on the N data processing tasks, N task nodes are created in the initial flowchart; Based on the input-output data relationship recorded in the data flow analysis results, directed edges are created between the N task nodes on the initial flowchart, wherein each directed edge connects two task nodes, and the direction of the directed edge indicates the data flow direction. When all the input-output data relationships recorded in the data flow analysis results are transformed into directed edges, the scheduling flowchart is obtained.

6. The scheduling method according to claim 1, characterized in that, Before uploading the N data processing tasks and the scheduling flowchart to the batch scheduling system, the process also includes: The scheduling flowchart and the task nodes are detected based on the batch task scheduling principle, and the detection results are obtained. When the detection result indicates that the scheduling flowchart and all the task nodes have passed the detection, the N data processing tasks and the scheduling flowchart are uploaded to the batch scheduling system.

7. The scheduling method according to claim 6, characterized in that, The steps for detecting the scheduling flowchart and the task nodes based on the batch task scheduling principle to obtain the detection results include: Check the circular dependencies of each task node in the scheduling flowchart to obtain the first check result; If the first check result indicates that the task node does not have a circular dependency, the task node is determined to have passed the test; The connectivity of the scheduling flowchart is checked to obtain a second check result; If the second check result indicates that the topology of the scheduling flowchart is fully connected, the scheduling flowchart is determined to have passed the check.

8. A scheduling device for data processing tasks, characterized in that, include: The acquisition unit is used to acquire the data processing mapping, wherein the data processing mapping refers to the mapping configuration file obtained after configuring the preset mapping template based on the data processing requirements; The generation unit is used to generate a data processing script based on the data processing mapping, and to generate N data processing tasks based on the data processing script, where N is a positive integer; A construction unit is used to construct a scheduling flowchart based on N data processing tasks. The scheduling flowchart includes N task nodes, each task node corresponds to one data processing task, and the scheduling flowchart is a directed acyclic graph used to record the execution order and dependencies of the data processing tasks. The upload unit is used to upload the N data processing tasks and the scheduling flowchart to the batch scheduling system, whereby the batch scheduling system performs task scheduling.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the scheduling method for the data processing task according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the scheduling method for the data processing task according to any one of claims 1 to 7.

Images