A task scheduling method, device, apparatus and storage medium

Abstract

The application discloses a task scheduling method and device, equipment and a storage medium, and belongs to the technical field of data warehouses. The method comprises the following steps: for each task read, acquiring various data sources of the task; determining a first mark value of the task according to a target value corresponding to each data source, wherein the target value represents a value of data update of the data source at the last time; acquiring a second mark value of the task, wherein the second mark value is a first mark value determined when the task is executed last time; comparing the first mark value with the second mark value to determine whether the data involved in the task has changed; and if the data involved in the task has changed, triggering execution of the task. The application can improve the timeliness of task execution and the use efficiency of computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data warehouse, and particularly relates to a task scheduling method and device, equipment and a storage medium. BACKGROUND

[0002] A data warehouse is a data collection for providing various types of support data for enterprise decision-making. Data mining or data analysis and other data processing on data in the data warehouse are completed by various tasks, and thus effectively scheduling and managing the tasks are very important work in data warehouse management.

[0003] The existing task scheduling methods mainly include manual trigger scheduling, time period scheduling and event trigger scheduling. The manual trigger scheduling needs to manually trigger the execution of a task job, and has low timeliness. The time period scheduling automatically starts the task job when the task reaches the time period, and cannot respond in time when data changes, and also executes when data does not change, thus causing repeated job problems. The event trigger scheduling introduces a workflow, configures event trigger conditions in each node of the workflow, and the execution of the task is passive, has low robustness, and the workflow configuration is complicated and has high complexity. SUMMARY

[0004] The present application provides a task scheduling method, device, equipment and storage medium to solve the technical problems of repeated job and low timeliness in the prior art.

[0005] To achieve the purpose, the technical solution adopted by the present application is:

[0006] On the one hand, the present application provides a task scheduling method, which comprises:

[0007] For each task read, obtaining each data source of the task;

[0008] According to the target value corresponding to each data source, determining the first mark value of the task, the target value representing the value of the data update of the data source last time;

[0009] Obtaining the second mark value of the task, the second mark value being the first mark value determined by the task last time;

[0010] Comparing the first mark value and the second mark value to determine whether the data involved in the task has changed;

[0011] If the data involved in the task has changed, triggering the execution of the task.

[0012] On the other hand, a task scheduling device is provided, which comprises:

[0013] a data source obtaining module, configured to obtain, for each task of reading, a respective data source of the task;

[0014] a first mark value obtaining module, configured to determine a first mark value of the task according to a target value corresponding to the respective data source, the target value representing a value of a latest data update of the data source;

[0015] a second mark value obtaining module, configured to obtain a second mark value of the task, the second mark value being the first mark value determined when the task was last executed;

[0016] a data change determining module, configured to compare the first mark value and the second mark value, and determine whether the data involved in the task has changed;

[0017] a task triggering module, configured to trigger execution of the task in a case where the data involved in the task has changed.

[0018] In another aspect, a task scheduling device is provided, the device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the task scheduling method as described above.

[0019] In another aspect, a computer storage medium is provided, the computer storage medium storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the task scheduling method as described above.

[0020] The task scheduling method, device, equipment and storage medium provided by the present application have the following beneficial effects:

[0021] By recording the data updates of the respective data sources, the task job execution is triggered only in a case where the data involved in the task has changed, thereby improving the timeliness of task execution; when the data does not change, the task job execution is not triggered, thereby avoiding occupying computing resources for repeated jobs and improving the use efficiency of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0023] Figure 1is a structural schematic diagram of a task scheduling system provided by an embodiment of the present application.

[0024] Figure 2 is a flow schematic diagram of a task scheduling method provided by an embodiment of the present application.

[0025] Figure 3 is a flow schematic diagram of determining a first mark value provided by an embodiment of the present application.

[0026] Figure 4 is a flow schematic diagram of constructing a metadata table provided by an embodiment of the present application.

[0027] Figure 5 is a flow schematic diagram of determining a dependency relationship tree provided by an embodiment of the present application.

[0028] Figure 6 is an example diagram of a dependency relationship tree provided by an embodiment of the present application.

[0029] Figure 7 is another flow schematic diagram of constructing a metadata table provided by an embodiment of the present application.

[0030] Figure 8 is an example of the constructed metadata table provided by an embodiment of the present application.

[0031] Figure 9 is another flow schematic diagram of constructing a metadata table provided by an embodiment of the present application.

[0032] Figure 10 is another example of the constructed metadata table provided by an embodiment of the present application.

[0033] Figure 11 is an example of obtaining a target value provided by an embodiment of the present application.

[0034] Figure 12 is another example of obtaining a target value provided by an embodiment of the present application.

[0035] Figure 13 is a flow schematic diagram of another task scheduling method provided by an embodiment of the present application.

[0036] Figure 14 is an example of updating a target value provided by an embodiment of the present application.

[0037] Figure 15 is an example flowchart of a task scheduling method provided by an embodiment of the present application.

[0038] Figure 16 is a structural schematic diagram of a task scheduling apparatus provided by an embodiment of the present application.

[0039] Figure 17 is a structural schematic diagram of a first marking value acquisition module provided by an embodiment of the present application.

[0040] Figure 18 is another structural schematic diagram of a first marking value acquisition module provided by an embodiment of the present application.

[0041] Figure 19 is a structural schematic diagram of a metadata table construction unit provided by an embodiment of the present application.

[0042] Figure 20 is a structural schematic diagram of a first construction unit provided by an embodiment of the present application.

[0043] Figure 21 is a structural schematic diagram of a second construction unit provided by an embodiment of the present application.

[0044] Figure 22 is another structural schematic diagram of a second construction unit provided by an embodiment of the present application.

[0045] Figure 23 is a structural schematic diagram of another task scheduling device provided by an embodiment of the present application.

[0046] Figure 24 is a structural schematic diagram of a data acquisition module provided by an embodiment of the present application.

[0047] Figure 25 is a hardware structural schematic diagram of a device for implementing the task scheduling method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] It should be noted that the terms "first", "second", and the like in the description and claims of the application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0050] First, the related terms involved in the embodiments of the application are explained as follows:

[0051] Data warehouse (DW): a subject-oriented, integrated, non-volatile, time-varying but relatively stable information data set, mainly used for data mining and data analysis, commonly known as Hive.

[0052] Metadata: also known as explanatory data or data dictionary, used to describe the data in the database, business data table and partition of the business data table in the data warehouse, such as the name of the database, the name of the business data table, the name of the partition of the business data table, the data update time and other custom description information, etc.

[0053] OLAP (Online Analytical Processing): by formulating the dimensions and indicators of data, submitting task jobs for operation, and obtaining multi-dimensional indicator analysis results.

[0054] Task: an operation configured when data mining or data analysis and other data applications are performed based on data in the data warehouse. Running the task will generate a job, and the execution of the job requires certain computing resources, such as CPU, memory, etc.

[0055] Task scheduling: a mechanism for running thousands of task jobs in the data warehouse.

[0056] ETL (Extract-Transform-Load): the process of loading data from business systems to data warehouse after extract, clean and transform, the purpose is to integrate the scattered, messy, non-uniform data in the enterprise together, and provide analysis basis for enterprise decision-making. Data acquisition from business systems into data warehouse, as well as data conversion and flow in data warehouse can be considered as the process of ETL.

[0057] Unix timestamp: a time representation, defined as the total number of seconds from January 1, 1970, 00:00:00 GMT (January 1, 1970, 08:00:00 Beijing time) to now.

[0058] Big data refers to the data collection that cannot be captured, managed and processed within a certain time range by conventional software tools, and is a large amount of information assets with high growth rate and diversity, which requires new processing mode to have stronger decision-making ability, insight discovery and process optimization ability. With the advent of the cloud era, big data has attracted more and more attention, and big data needs special technology to effectively process large amount of data within a certain time range. The technologies suitable for big data include large-scale parallel processing database, data mining, distributed file system, distributed database, cloud computing platform, Internet and scalable storage system.

[0059] In the field of big data, the common ways of task scheduling for data warehouse data are manual trigger scheduling, time period scheduling, event trigger scheduling, etc.

[0060] Manual trigger scheduling means that each execution of the task job is triggered manually. When the data is updated, the task responsible person needs to perceive and trigger the task job execution, which is low in timeliness and cumbersome to operate.

[0061] Time period scheduling sets time period scheduling rules for the task. The task scanning module judges that the task of this type of scheduling mode reaches the time trigger condition, and automatically starts a task job. For example, it is triggered once an hour, triggered once at a fixed time (such as 00:00) every day, etc. The execution of the task job is a fixed time. If the time is set too long, when the data changes, the task job cannot be executed in time, which is low in timeliness; if the time is set too short, when the data does not change, the task job will still be executed when the time trigger condition is met, which causes repeated job problems and reduces the effective use of computing resources.

[0062] The event-triggered scheduling configures a series of serial / parallel tasks by introducing a workflow, defines event-triggered conditions of each node, and triggers task execution by receiving events when the workflow is started. The task is a node in the workflow, and the execution of the task is triggered by the event generated after the execution of the previous node. The execution of each link task is triggered by the front and rear dependent manner, and the configuration is relatively cumbersome. The event-triggered type is a passive execution mode, and the execution of the task job depends on the occurrence of the event. If the event itself is lost, the task will not be executed. In addition, if the task runs incorrectly, the task job will not be triggered to execute again without retry logic, and the robustness is not high.

[0063] Unlike the existing task scheduling mode, the embodiment of the present application provides a task scheduling method based on data changes. The execution of the task job is triggered only when the data involved in the task changes.

[0064] Please refer to Figure 1 , which is a structural schematic diagram of a task scheduling system provided by the embodiment of the present application. As Figure 1 indicated, the task scheduling system includes a data acquisition module 11, a data storage module 12, a task scanning module 13, and a task execution module 14.

[0065] The data acquisition module 11 is configured to load source data of a business system to a database 122 of the data storage module 12 through ETL, and update metadata corresponding to the database 122 in a metadata database 121. The updated metadata includes a target value representing the latest data update, and the type of the target value is any type that can be used for size comparison, including but not limited to Unix timestamp such as millisecond and second. The task scanning module 13 is configured to scan a pre-established task sequence, read each task in the task sequence, obtain a target value matched with the task from the data storage module 12, determine whether the data involved in the task changes, and send a task execution request to the task execution module 14 in the case that the data involved in the task changes. The task execution module 14 receives the task execution request, triggers the execution of the task, and updates the data time of the task after the execution of the task is completed. The type of the data time of the task is consistent with the type of the target value.

[0066] Specifically, the data collection module 11 includes a data collection unit 111 and a metadata updating unit 112. The data collection unit 111 is configured to collect source data of a business system, extract and clean the source data to obtain target data, and save the target data into a corresponding business data table or a partition of the business data table in a database 122. The metadata updating unit 112 is configured to store metadata corresponding to the business data table or metadata corresponding to a partition of the business data table in a metadata database 121. It can be understood that, in some embodiments, the database 122 and the metadata database 121 can be the same database, and the data storage module 12 can also include a plurality of databases 122. Each database 122 includes at least one business data table, and each business data table is configured to store actual business data. Typically, the database 122 can be a database such as Hive. The metadata database 121 can include a plurality of metadata tables, and each metadata table includes a plurality of metadata. Typically, an additional fact relational database is used as the metadata database 121, such as MySQL.

[0067] Specifically, the task scanning module 13 includes a task reading unit 131, a metadata reading unit 132, and a data change determining unit 133. After the data collected by the data collection module 11 enters the database 122 of the data warehouse, a series of tasks, i.e., a task sequence, are created based on the data stored in the database 122 when data analysis or data application based on the data stored in the database 122 is performed, and the task sequence is stored in the database 122. The task sequence is polled and read by the task reading unit 131, and the read task is sent to the data change determining unit 133, or is sent to the data change determining unit 133 after it is detected that the read task is in a ready state. After receiving the task, the data change determining unit 133 reads a target value corresponding to a data source of the task from the metadata reading unit 132, determines a latest data update time of the data source according to the read target value, compares the latest data update time with a data time of the task, and determines whether the data involved in the task has changed. In the case where the data involved in the task has changed, a task execution request is sent to the task execution module 14.

[0068] Specifically, the task execution module 14 includes a job execution unit 141 and a task updating unit 142. After receiving the task execution request from the data change determining unit 133, the job execution unit 141 reads data in a data source involved in the task, and executes a specific data job. After the data job is completed, the task updating unit 142 updates metadata corresponding to the data source involved in the task. In actual applications, OLAP is often used to analyze complex data to obtain intuitive and easy-to-understand query results.

[0069] When ETL collection is performed on a business data table in a data warehouse, the target value corresponding to the data source is recorded. Then, for each scanned task, it can be determined according to the recorded target value whether the data involved in the task has changed, and in the case where the data has changed, the execution of the task job is triggered. Compared with the existing manual trigger scheduling, time period scheduling and event trigger scheduling mode, the task job trigger is more timely, and at the same time, it can also avoid repeated jobs (useless work jobs) when the data does not change, so as to improve the data calculation efficiency, improve the use efficiency of operation resources, and make the task execution more robust.

[0070] The following introduces a task scheduling method of the present application, Figure 2 is a flowchart of a task scheduling method provided by an embodiment of the present application. The present specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of the many execution orders of the steps, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment). Specifically, as shown in Figure 2 The method can include:

[0071] S201, for each task read, obtaining each data source of the task.

[0072] In an embodiment of the present application, each data source includes at least one database, each database includes at least one business data table, and each business data table can be a partitioned business data table or a non-partitioned business data table. The non-partitioned business data table refers to a corresponding business data table established in a non-partitioned mode, and the partitioned business data table refers to a corresponding business data table established in a partitioned mode. If the business data table is a partitioned business data table, the business data table includes at least one partition.

[0073] Generally, when a business data table is established in a data warehouse, a partitioned mode is used to improve the query speed of data. In the partitioned mode, the business data table is managed by partitioning. The partitioning method generally adopts time partitioning, such as day partitioning, day / hour partitioning, etc. Taking day / hour partitioning as an example, if the value is 2020060120, it represents the data partition of 20:00 on June 1, 2020, and if the partitioned business data table includes N partitions, each partition corresponds to a different day / hour.

[0074] When the target data enters the database in the data warehouse, a task sequence is created based on the target data in the database for data analysis or data application. When the task sequence is created, task information of each task in the task sequence is configured, which can include task ID, task name, task status, data source, data source identifier, data time of the task, etc. In specific implementation, the configuration of the task information can be set as shown in the following table, and the content of the task information is not limited to the content shown in the table, and can be defined by the task as needed.

[0075]

[0076] For each task in the task sequence, in order to avoid repeated execution, the state of the task can be judged by the task state in the task information, and only the task in the ready state can be executed. The ready state is that the last job of the task is in the completed state. Therefore, before the task obtains each data source, the step of detecting whether the task is in the ready state is further included.

[0077] Specifically, the detection of whether the task is in the ready state includes: obtaining the task state of the task; if the task state is not the end state, the task is not in the ready state; if the task state is the end state, the task is in the ready state.

[0078] S202, according to the target value corresponding to each data source, determining a first mark value of the task, the target value representing a value of the data update of the last time of the data source.

[0079] The type of the target value can be any type that can be used for size comparison, for example, it can be a data update time (such as a timestamp of seconds, milliseconds, etc.), or an Arabic numeral stored in an incremental manner, etc. In some embodiments, the target value can also be a date or other type that can represent the early or late data update.

[0080] The data warehouse itself is a data set created for the purpose of analytic reporting and decision support, and metadata describing the target data in the data warehouse is stored for the purpose of quickly supporting analytic reporting and decision support related tasks. There are various ways to store the metadata, such as using XML, excel, or other configuration files, or directly storing in a field of a corresponding business data table. Understandably, when the data capacity stored by the data warehouse is large, the amount of metadata to be stored by the configuration file will also increase. When obtaining the metadata, it is necessary to filter from a large amount of data, which will seriously affect the efficiency of obtaining, and the configuration file storage method has low flexibility for metadata management. Directly storing the metadata in the corresponding business data table will separate the business from the application, and is also a disadvantage for data management.

[0081] To improve the efficiency of obtaining, enhance the flexibility of data management, and separate the business from the application, the embodiment of the application stores the metadata corresponding to the database, the metadata corresponding to the business data table, and the metadata corresponding to the partition of the business data table by establishing a separate metadata table.

[0082] Based on different storage methods, with reference to Figure 3 The first mark value of the task is determined according to the target value corresponding to each data source, and includes:

[0083] S2021, determining the storage method of the target value corresponding to each data source.

[0084] S2022, if the storage method of the target value is metadata table storage, obtaining the target value corresponding to the data source by querying the metadata table corresponding to the data source.

[0085] S2023, if the storage method of the target value is non-metadata table storage, obtaining the target value corresponding to the data source by querying the non-metadata table corresponding to the data source.

[0086] S2024, determining the maximum value in the target values corresponding to each data source obtained by querying as the first mark value.

[0087] The storage method of the target value is the storage method of the metadata. If the storage method of the metadata is metadata table storage, in a preferred embodiment, before obtaining the target value corresponding to the data source by querying the metadata table corresponding to the data source, the step of constructing the metadata table corresponding to the data source is further included. With reference to Figure 4 The metadata table corresponding to the data source is constructed, and includes:

[0088] S401, construct a dependency tree corresponding to each database in the data source.

[0089] The data source involved in a task can be the same database or different databases, and each database includes at least one business data table. In order to construct the metadata table corresponding to the data source, it is necessary to determine the dependency tree based on the dependency relationship between the business data table and the database in the data source, and to construct the metadata table corresponding to the data source based on the dependency tree. As shown in Figure 5 The step 401 can include:

[0090] S4011, determining the root node of the dependency tree according to the database.

[0091] S4012, determining the first child node corresponding to each business data table in the database according to the database, and the first child node is subordinate to the root node.

[0092] S4013, for any one business data table in the database, if the business data table is a partitioned business data table, then determining the second child node corresponding to the partition in the business data table according to the partition, and the second child node is subordinate to the first child node corresponding to the business data table.

[0093] Since the business data table can be constructed in a partitioned mode, i.e. a partitioned business data table, or in a non-partitioned mode, i.e. a non-partitioned business data table. Correspondingly, the dependency tree corresponding to the database includes a first type and a second type, the first type corresponds to all business data tables being partitioned business data tables, and the second type corresponds to at least one business data table being a non-partitioned business data table. Then, the dependency tree corresponding to the first type includes a root node and a first child node, and the dependency tree corresponding to the second type includes a root node, a first child node and a second child node. The root node is the database, the first child node is the business data table, and the second child node is the partition.

[0094] For example, database 1 includes business data table a and business data table b, and business data table a and business data table b are both non-partitioned business data tables, then the dependency tree corresponding to database 1 includes a root node, first child node a and first child node b. The root node is database 1, the first child node a is business data table a, the first child node b is business data table b, and the first child node a and the first child node b are both subordinate to the root node. If the business data table a is a partitioned business data table as described above, and the business data table a includes partition a1 and partition a2, then the dependency tree corresponding to the database 1 further includes second child node a1 and second child node a2; the second child node a1 is the partition a1, the second child node a2 is the partition a2, and the second child node a1 and the second child node a2 are subordinate to the first child node a.

[0095] The structure of the dependency tree can refer to that shown in Figure 6 In Figure 6 , the dependency tree 1 is of the first kind, the dependency tree 2 and the dependency tree 3 are of the second kind, the dependency tree 1 corresponds to each business data table in the database being a non-partitioned business data table, the dependency tree 2 corresponds to only one business data table in the database being a partitioned business data table, and the dependency tree 3 corresponds to all business data tables in the database being partitioned business data tables.

[0096] S402, constructing a metadata table corresponding to the data source according to each dependency tree.

[0097] In the case of the dependency tree being of the first kind, each dependency tree only includes a root node and a first child node, and then the meta-database includes a database metadata table, a first metadata table and a first metadata description table.

[0098] Specifically, as shown in Figure 7 , the constructing a metadata table corresponding to the data source according to each dependency tree comprises:

[0099] S4021, constructing a database metadata table according to all root nodes in the dependency trees, the database metadata table taking the unique identifier of the root node as the primary key.

[0100] The database metadata table (DATABASES) is used to store metadata information of the database, such as the name of the database (DB_NAME) and the like. Each row in the DATABASES corresponds to a different database, and the unique identifier of the root node is the unique identifier of the database (DB_ID), and different databases are identified by the DB_ID.

[0101] S4022, constructing a first metadata table according to all first child nodes in the dependency trees, the first metadata table taking the unique identifier of the root node corresponding to the first child node and the unique identifier of the first child node as the joint primary key.

[0102] The first metadata table (TABLES) is used to store metadata information of the business data table, such as the name (TABLE_NAME) of the business data table. Each row in the TABLES corresponds to a different business data table, and the unique identifier of the first child node is the unique identifier (TABLE_ID) of the business data table, which is associated with the DATABASES through the DB_ID. The relationship between the DATABASES and the TABLES is 1:N, that is, one DB_ID can correspond to N records in the TABLES, and N depends on the number of business data tables. The TABLES includes records of business data tables from different databases, and each record in the TABLES can be uniquely identified by the DB_ID and the TABLE_ID.

[0103] S4023, constructing a first metadata description table based on the first metadata table, the first metadata description table taking the unique identifier of the first child node and the unique identifier of the first target value as a joint primary key, and the first target value being a target value corresponding to the business data table corresponding to the first child node.

[0104] The first metadata description table (TABLE_PARAMS) is used to store custom description information of the business data table, such as the number of data rows of the business data table, the data update time, and the like. The TABLE_PARAMS is associated with the TABLES through the TABLE_ID, and each item of description information is stored in the form of key-value. All custom description information of the corresponding business data table can be obtained through the TABLE_ID, and then the corresponding target value (PARAM_VALUE) can be obtained through the unique identifier of the first target value (PARAM_KEY). For ease of description, the metadata in the TABLE_PARAMS for describing the target value is defined as the data table metadata, and the association relationship between the metadata tables can be referred to as shown in Figure 8

[0105] In the case of the second type of dependency relationship tree, each dependency relationship tree includes a second child node in addition to the root node and the first child node, and the meta database includes a second metadata table and a second metadata description table in addition to the database metadata table, the first metadata table, and the first metadata description table.

[0106] Correspondingly, according to the method for constructing the metadata table corresponding to the data source according to each dependency relationship tree, the method further includes: Figure 9

[0107] S4024, constructing a second metadata table according to all second child nodes in the dependency relationship trees, the second metadata table taking the unique identifier of the first child node corresponding to the second child node and the unique identifier of the second child node as a joint primary key. ​​

[0108] The second metadata table (PARTITIONS) is used to store partition metadata information of the business data table, such as the name of the partition (PART_NAME). Each row in the PARTITIONS corresponds to a different partition, and the unique identifier of the second child node is the unique identifier of the partition (PART_ID). Different partitions are identified by the PART_ID, and are associated with the TABLE_ID in the TABLES. The relationship between the TABLES and the PARTITIONS is 1:N, that is, one TABLE_ID corresponds to N records in the PARTITIONS, and N depends on the number of partitions. The PARTITIONS includes records of partitions from different business data tables, and each record in the second metadata table can be uniquely identified by the TABLE_ID and the PART_ID.

[0109] In S4025, a second metadata description table is constructed based on the second metadata table, and the second metadata description table takes the unique identifier of the second child node and the unique identifier of the first target value as a joint primary key.

[0110] The second metadata description table (PARTITION_PARAMS) is used to store partition custom description information of the business data table, such as the data update time of the partition. Each item of description information is stored in the form of key-value. All custom description information of the corresponding partition can be obtained by the PART_ID, and then the corresponding target value (PARAM_VALUE) can be obtained by the unique identifier of the first target value (PARAM_KEY). In order to facilitate description, the metadata in the PARTITION_PARAMS for describing the target value is defined as partition metadata, and the association relationship between the metadata tables can be referred to in the description of the first metadata table. Figure 10

[0111] It should be noted that, Figure 8 and Figure 10 are only examples. In some embodiments, other fields can be defined in the metadata table according to the needs of data description, and the type and length of the fields are defined, which are not limited in the embodiments of the present application.

[0112] ​After the metadata table corresponding to the data source is constructed, if the storage mode of the target value is metadata table storage, the data source identifier corresponding to the data source can be obtained, a metadata table query statement is constructed, and the target value corresponding to the data source is obtained. The data source identifier at least includes a database name (DB_NAME) and a business data table name (TABLE_NAME). If the data source further includes multiple partitions, the data source identifier should further include the name (PART_NAME) of each partition. The partitions involved in each task can be continuous or not continuous. For continuous partitions, in order to reduce the complexity of parameter configuration, only the name of the starting partition (START_PART_NAME) and the name of the ending partition (END_PART_NAME) can be set in the task information.

[0113] For ease of description, the query mode of the target value is exemplarily illustrated by taking the data update time as the target value, and the data update time is a Unix timestamp in milliseconds.

[0114] In the first step, the unique identifier (TABLE_ID) of the business data table is obtained from DATABASES and TABLES according to the database name (DB_NAME) and the business data table name

[0115] (TABLE_NAME), and the SQL (Structured Query Language) is as follows:

[0116] SELECT TABLES.TABLE_ID AS TABLE_ID

[0117] FROM DATABASES

[0118] INNER JOIN TABLES

[0119] ON DATABASES.DB_ID=TABLES.DB_ID

[0120] WHERE DATABASES.NAME=${DB_NAME}

[0121] AND TABLES.TABLE_NAME=${TABLE_NAME}.

[0122] In the second step, the first mark value is obtained according to the obtained TABLE_ID.

[0123] In acquiring the first mark value, it is necessary to distinguish whether the task is a full task or an incremental task. The full task is to read all partitions of the business data table each time, and the incremental task is to read only one or more partitions of the data each time. If it is a full task, since the business data table has its corresponding data table metadata (metadata storing data update time), the value of the data table metadata can be directly used as the first mark value; if it is an incremental task, the maximum value in the partition metadata (metadata storing data update time) corresponding to the one or more partitions needs to be used as the first mark value.

[0124] If the PARAM_KEY of the storage data table metadata and the partition metadata is defined as LastDdlTime, for the full task, as shown in the following formula, the PARAM_VALUE corresponding to the PARAM_KEY LastDdlTime in the first metadata description table (TABLE_PARAMS) is the first mark value. The reference SQL is as follows: Figure 11

[0125] SELECT PARAM_KEY,PARAM_VALUE

[0126] FROM TABLE_PARAMS

[0127] WHERE TABLE_ID=${TABLE_ID}

[0128] AND PARAM_KEY='LastDdlTime'.

[0129] For the incremental task, as shown in the following formula, if the multiple partitions are continuous partitions, according to the name of the starting partition (START_PART_NAME) and the name of the ending partition (END_PART_NAME), the custom parameter aggregation value of the business data table involved partition is calculated from the second metadata table (PARTITIONS) and the second metadata description table (PARTITION_PARAMS) using the MAX aggregation function. The AGG_VALUE of the PARAM_KEY LastDdlTime is the first mark value. The reference SQL is as follows: Figure 12

[0130] SELECT PARTITION_PARAMS.PARAM_KEY AS PARAM_KEY,MAX(PARTITION_PARAMS.PARAM_VALUE)AS AGG_VALUE

[0131] FROM PARTITIONS

[0132] ​​INNER JOIN PARTITION_PARAMS

[0133] ON PARTITIONS.PART_ID = PARTITION_PARAMS.PART_ID

[0134] WHERE PARTITIONS.TABLE_ID = ${TABLE_ID}

[0135] AND PARTITIONS.PART_NAME >= ${START_PART_NAME}

[0136] AND PARTITIONS.PART_NAME <= ${END_PART_NAME}

[0137] GROUP BY PARTITION_PARAMS.PARAM_KEY.

[0138] It should be noted that if the multiple partitions are not continuous partitions, the idea of obtaining the first mark value is the same as that of the above continuous partitions, and will not be repeated here.

[0139] S203, obtain a second mark value of the task, the second mark value being a first mark value determined by the task in the last execution.

[0140] Each task will update the data time of the task with the first mark value determined by the task in the execution.

[0141] S204, compare the first mark value and the second mark value, and determine whether the data involved in the task has changed.

[0142] Specifically, it is judged whether the second mark value is greater than the first mark value; if the second mark value is greater than the first mark value, the data involved in the task has changed; if the second mark value is less than or equal to the first mark value, the data involved in the task has not changed.

[0143] S205, if the data involved in the task has changed, trigger execution of the task.

[0144] After receiving the job execution request from the data change judgment unit, the job execution unit reads the data from the relevant data source and executes the specific data job. From the start of the job until its completion, the task is in a "running" state. Tasks in this state are not scheduled or judged by the task scanning module to avoid duplicate jobs. After the job is completed, it enters a "completed" state, where tasks can be scheduled and judged by the task scanning module. After the data job is completed, the task's data time is updated by the task update unit. This data time is the first marker value determined during the current task execution. If the task job fails to execute successfully, such as by throwing an exception during execution, the task's data time is not updated, and the task will be scanned again to improve the robustness of task execution.

[0145] The data stored in the database of the data warehouse is collected from business systems through ETL. Each task is also created based on the data stored in the database. Therefore, referring to... Figure 13 As shown, in some embodiments, before obtaining the respective data sources for each reading task, the method further includes:

[0146] S1301, in response to a service collection request, source data from the service system is collected, wherein the service collection request includes a collection mode.

[0147] S1302, After extracting, cleaning and transforming the source data, the target data is obtained.

[0148] S1303, Based on the acquisition mode, the target data is written to each data source involved in the service acquisition request.

[0149] S1304, Update the target values ​​corresponding to each of the data sources.

[0150] Data acquisition units generally operate in two modes when collecting data: full acquisition mode and incremental acquisition mode. In full acquisition mode, all data is collected from the business system. First, old data in the business data table or all partitions of the business data table is deleted. Then, the newly acquired target data is written to the business data table or its corresponding partition. In incremental acquisition mode, data is incrementally collected from the business system, and then the target data is appended to the business data table or its corresponding partition.

[0151] When updating the target values ​​corresponding to various data sources, if the target values ​​are stored in a non-metadata table, such as Excel or XML, the target values ​​in the non-metadata table corresponding to each data source can be updated directly. There are many solutions for updating data in non-metadata tables such as Excel and XML in the existing technology, which will not be elaborated here.

[0152] If the storage mode of the target value is metadata table storage, it is necessary to distinguish whether the business data table in the data source is a partitioned business data table or a non-partitioned business data table. If the business data table is a non-partitioned business data table, the target value corresponding to the business data table is directly updated. If the business data table is a partitioned business data table, the target values corresponding to each partition (or the involved partitions) in the business data table are first updated, and then the maximum value among the target values corresponding to all partitions (or all involved partitions) in the business data table is used to update the target value corresponding to the business data table.

[0153] For example, as shown in FIG. 6, if the data source involved in ETL is N-M+1 continuous partitions from partition M to partition N (N is greater than M) of a business data table, after writing data of each partition, the target value corresponding to each partition is updated, and after completing the data writing of N-M+1 partitions, the maximum value among the target values corresponding to the N-M+1 partitions is taken as the target value corresponding to the business data table. Figure 14 Figure 14 In the example shown in FIG. 6, the target value is stored in the form of key-value pair, the key defines LastDdlTime, and the value is defined as Unix timestamp in milliseconds. Figure 14 As shown in FIG. 6, the target value corresponding to partition M is 1591782610000, the target value corresponding to partition M+1 is 1591782620000, the target value corresponding to partition N is 1591782630000, and the target values (not shown in the figure) corresponding to other partitions are less than 1591782630000, so 1591782630000 is used as the target value corresponding to the business data table.

[0154] The following takes the data update time as the target value, and each business data table as a partitioned business data table as an example to further illustrate the key steps of the task scheduling method of the embodiments of the present application. As shown in FIG. 7, the method comprises the following steps. Figure 15

[0155] S1501, ETL collection is performed on each partition of each business data table in the data warehouse, and the data update time of each partition of the business data table and the data update time of the business data table are recorded in the metadata table.

[0156] As shown in FIG. 8, after ETL collection, the data update time can be written or updated into TABLE_PARAMS and PARTITION_PARAMS. PARAM_KEY is defined as LastDdlTime, and PARAM_VALUE takes the value of Unix timestamp in milliseconds. Figure 14

[0157] ​​​S1502, scan task, data reading time of the task, and data update time of each partition of the business data table involved in the task.

[0158] Continue to refer to Figure 14 As shown, if the task involves N-M+1 consecutive partitions of the business data table from partition M to partition N, then N-M+1 data update times are obtained.

[0159] S1503 determines whether the data involved in the task has been updated based on the data update time of each partition and the data time of the task.

[0160] Take the maximum value among N-M+1 data update times, such as Figure 14 As shown, the maximum data update time is 1591782630000. If the data time stored in the task is less than 1591782630000, it indicates that the data involved in the task has been updated, and step S1504 is executed to trigger the task job execution. If the data time stored in the task is greater than or equal to 1591782630000, it indicates that the data involved in the task has not been updated, and the task is not executed; the process returns to step S1502. In the next scan cycle, the task will be rescanned.

[0161] S1504 triggers task execution; after completion, the task's data time is updated.

[0162] When ETL is performed on the business data tables of the data warehouse, the data update time of the business data tables and their partitions is recorded in the metadata table. The task scanning module performs a polling scan on the pre-established task sequence, comparing the data time of the ready tasks with the data update times of the partitions of the business data tables involved in the task. If the data update times of all partitions of the business data tables are less than the data time of the task, it indicates that the data involved in the task has not been updated; otherwise, the data involved in the task has been updated, triggering the task job execution. After the task job is completed, the maximum value of the data update times of all partitions of the involved business data tables is taken, and the task's data time is updated accordingly.

[0163] From the technical solutions provided by the embodiments of the present application, it can be seen that the embodiments of the present application record the data updates of each data source, so that the task job execution is triggered only when it is determined that the data involved in the task has changed, thereby improving the timeliness of task execution; when the data does not change, the task job execution is not triggered, thereby avoiding occupying computing resources for repeated jobs and improving the use efficiency of computing resources; the metadata is stored using a separate metadata table, so that the analysis and the business are separated, the data query rate and the data analysis efficiency can be improved; only when the task job is successfully executed, the data time of the task is updated, and if an error occurs during task execution, the task will be scanned and triggered again, thereby improving the robustness of task execution.

[0164] The embodiments of the present application also provide a task scheduling apparatus, as shown in Figure 16 The apparatus can include:

[0165] A data source obtaining module 161 is configured to obtain each data source of a task for each read task;

[0166] A first mark value obtaining module 162 is configured to determine a first mark value of the task according to a target value corresponding to each data source, the target value representing a value of a data update of the data source at the last time;

[0167] A second mark value obtaining module 163 is configured to obtain a second mark value of the task, the second mark value being a first mark value determined by the task at the last execution;

[0168] A data change determining module 164 is configured to compare the first mark value and the second mark value, and determine whether the data involved in the task has changed;

[0169] A task triggering module 165 is configured to trigger the execution of the task when the data involved in the task has changed.

[0170] In some embodiments, referring to Figure 17 The first mark value obtaining module 162 can include:

[0171] A storage mode obtaining unit 1621 is configured to determine the storage mode of the target value corresponding to each data source;

[0172] A metadata table querying unit 1622 is configured to obtain the target value corresponding to the data source by querying the metadata table corresponding to the data source when the storage mode of the target value is metadata table storage;

[0173] The non-metadata table query unit 1623 is configured to, in a case where the storage mode of the target value is a non-metadata table storage, query the non-metadata table corresponding to the data source to obtain the target value corresponding to the data source.

[0174] The first mark value determination unit 1624 is configured to determine the maximum value in the target values corresponding to the respective data sources as the first mark value.

[0175] In some embodiments, as shown in FIG. 1, Figure 18 The first mark value acquisition module 162 can further include:

[0176] The metadata table construction unit 1625 is configured to construct the metadata table corresponding to the data source.

[0177] Specifically, as shown in FIG. 1, Figure 19 The metadata table construction unit 1625 can include:

[0178] The first construction unit 16251 is configured to construct the dependency relationship tree corresponding to each database in the data source;

[0179] The second construction unit 16252 is configured to construct the metadata table corresponding to the data source according to the respective dependency relationship trees.

[0180] Specifically, as shown in FIG. 1, Figure 20 The first construction unit 16251 can include:

[0181] The root node construction unit 162511 is configured to determine the root node of the dependency relationship tree according to the database;

[0182] The first child node construction unit 162512 is configured to determine the first child node corresponding to each business data table in the database according to the business data table, and the first child node is subordinate to the root node;

[0183] The second child node construction unit 162513 is configured to, for any one business data table in the database, if the business data table is a partitioned business data table, determine the second child node corresponding to the partition in the business data table according to the partition, and the second child node is subordinate to the first child node corresponding to the business data table.

[0184] Specifically, as shown in FIG. 1, Figure 21 The second construction unit 16252 can include:

[0185] The database metadata table construction unit 162521 is configured to construct the database metadata table according to all the root nodes in the respective dependency relationship trees, and the database metadata table takes the unique identifier of the root node as the primary key.

[0186] The first metadata table construction unit 162522 is configured to construct a first metadata table according to all first child nodes in the respective dependency relationship trees, the first metadata table taking the unique identifier of the root node corresponding to the first child node and the unique identifier of the first child node as joint primary keys;

[0187] The first metadata description table construction unit 162523 is configured to construct a first metadata description table based on the first metadata table, the first metadata description table taking the unique identifier of the first child node and the unique identifier of a first target value as joint primary keys, the first target value being a target value corresponding to the business data table corresponding to the first child node.

[0188] In some embodiments, as shown in Figure 22 The second construction unit 16252 can further include:

[0189] The second metadata table construction unit 162524 is configured to construct a second metadata table according to all second child nodes in the respective dependency relationship trees, the second metadata table taking the unique identifier of the first child node corresponding to the second child node and the unique identifier of the second child node as joint primary keys;

[0190] The second metadata description table construction unit 162525 is configured to construct a second metadata description table based on the second metadata table, the second metadata description table taking the unique identifier of the second child node and the unique identifier of a first target value as joint primary keys.

[0191] In some embodiments, as shown in Figure 23 The apparatus can further include:

[0192] The data collection module 166 is configured to collect data from a business system and update a database or a metadata database in a data warehouse according to the collected data.

[0193] Specifically, as shown in Figure 24 The data collection module 166 can include:

[0194] The source data collection unit 1661 is configured to collect source data of a business system in response to a business collection request, the business collection request including a collection mode.

[0195] The source data processing unit 1662 is configured to obtain target data by extracting and cleaning and converting the source data.

[0196] The data storage unit 1663 is configured to write the target data into respective data sources involved in the business collection request based on the collection mode.

[0197] The metadata updating unit 1664 is configured to update the target value corresponding to each data source.

[0198] It should be noted that the apparatus provided by the above-mentioned embodiments, when realizing the functions thereof, only takes the above-mentioned division of each functional module as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to the needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided by the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0199] The application also provides a task scheduling device, which is characterized in that the device comprises a processor and a memory, the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to realize the task scheduling method provided by the above-mentioned method embodiments.

[0200] Further, Figure 25 A hardware structure schematic diagram of a device for realizing the task scheduling method provided by the embodiments of the application is shown, and the device can participate in constituting or containing the task scheduling apparatus provided by the embodiments of the application. As shown in Figure 25 The device 250 can include one or more (in the figure, 2502a, 2502b, …, 2502n are used to show) processors 2502 (the processor 2502 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 2504 for storing data, and a transmission device 2506 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 25 The structure shown is only schematic, and it does not limit the structure of the above-mentioned electronic device. For example, the device 250 can also include more or fewer components than those shown in Figure 25 or have a different configuration from that shown in Figure 25 .

[0201] It should be noted that the one or more processors 2502 and / or other data processing circuitry described above can be referred to herein generically as "data processing circuitry". The data processing circuitry can be embodied as software, hardware, firmware, or any combination thereof, in whole or in part. Moreover, the data processing circuitry can be a single independent processing module or any combination of all or part of other elements included in the device 250 (or mobile device). As referred to in the embodiments of the present application, the data processing circuitry serves as a processor to control, for example, selection of a variable resistance terminal path connected to an interface.

[0202] The memory 2504 can be used to store software programs and modules of application software, and program instructions / data storage means corresponding to the method described in the embodiments of the present application. The processor 2502 can execute various functional applications and data processing by running the software programs and modules stored in the memory 2504, i.e., implement the task scheduling method described above. The memory 2504 can include a high-speed random access memory, and can further include a non-volatile memory such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 2504 can further include a memory remotely arranged with respect to the processor 2502, which can be connected to the device 250 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0203] The transmission device 2506 is configured to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the device 250. In one example, the transmission device 2506 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one embodiment, the transmission device 2506 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0204] The display can be, for example, a touch screen type liquid crystal display (LCD) which can enable a user to interact with a user interface of the device 250 (or mobile device).

[0205] The embodiments of the present application also provide a computer storage medium which can be arranged in a device to save at least one instruction or at least one program related to a task scheduling method in the method embodiments, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the task scheduling method provided by the above method embodiments.

[0206] Optionally, in the embodiment, the storage medium can be located in at least one of the plurality of network servers of the computer network. Optionally, in the embodiment, the storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing program codes.

[0207] From the above embodiments of the task scheduling method, device, equipment and storage medium provided by the present application, it can be seen that the embodiments of the present application determine whether the data involved in the task has changed according to the recorded target value for each task read during data collection, and only in the case that the data has changed, the task job execution is triggered, thereby improving the timeliness of task execution; when the data does not change, the task job execution is not triggered, thereby avoiding occupying the computing resources for repeated jobs and improving the use efficiency of computing resources; the metadata is stored in a separate metadata table, which separates the analysis from the business, and can improve the data query rate and data analysis efficiency; only when the task job is successfully executed, the data time of the task is updated, and if an error occurs during task execution, the task will be scanned and triggered again, thereby improving the robustness of task execution.

[0208] It should be noted that the above-mentioned order of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And the above describes the specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from the order in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or can be advantageous.

[0209] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, the device and equipment embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0210] The above description has fully disclosed the specific embodiments of the present application. It should be noted that any modification of the specific embodiments of the present application made by those skilled in the art does not deviate from the scope of the claims of the present application. Accordingly, the scope of the claims of the present application is not limited to the above specific embodiments.

Claims

1. A task scheduling method, characterized by, The method comprises: For each task read, obtaining each data source of the task; According to the target value corresponding to each data source, determining the first mark value of the task, the target value representing the value of the last data update of the data source; Obtain the second mark value of the task, the second mark value being the first mark value determined by the task in the last execution; Compare the first mark value and the second mark value to determine whether the data involved in the task has changed; If the data involved in the task has changed, trigger the execution of the task; According to the target value corresponding to each data source, determining the first mark value of the task, comprises: According to each data source, determine the storage mode of its corresponding target value; If the storage mode of the target value is metadata table storage, the target value corresponding to the data source is obtained by querying the metadata table corresponding to the data source; If the storage mode of the target value is non-metadata table storage, the target value corresponding to the data source is obtained by querying the non-metadata table corresponding to the data source; The maximum value of the target value corresponding to each data source obtained by querying is determined as the first mark value.

2. The method of claim 1, wherein, Each data source includes at least one database, and the database includes at least one business data table. If the storage mode of the target value is metadata table storage, before the target value corresponding to the data source is obtained by querying the metadata table corresponding to the data source, the step of constructing the metadata table corresponding to the data source is further included; The construction of the metadata table corresponding to the data source comprises: Constructing a dependency relationship tree corresponding to each database in the data source; According to each dependency relationship tree, constructing the metadata table corresponding to the data source.

3. The method of claim 2, wherein, The construction of the dependency relationship tree corresponding to each database in the data source comprises: According to the database, determining the root node of the dependency relationship tree; According to each business data table in the database, determining a first child node corresponding to each business data table, the first child node being subordinate to the root node; For any one business data table in the database, if the business data table is a partitioned business data table, according to the partition in the business data table, determining a second child node corresponding to the partition, the second child node being subordinate to the first child node corresponding to the business data table.

4. The method according to claim 3, characterized in that, According to each dependency relationship tree, constructing the metadata table corresponding to the data source, comprises: According to all root nodes in each dependency relationship tree, constructing a database metadata table, the database metadata table taking the unique identifier of the root node as the primary key; According to all first child nodes in each dependency relationship tree, constructing a first metadata table, the first metadata table taking the unique identifier of the root node corresponding to the first child node and the unique identifier of the first child node as the joint primary key; Based on the first metadata table, constructing a first metadata description table, the first metadata description table taking the unique identifier of the first child node and the unique identifier of the first target value as the joint primary key, the first target value being the target value corresponding to the business data table corresponding to the first child node.

5. The method according to claim 4, characterized in that, The method further comprises: constructing a second metadata table according to all second child nodes in each of the dependency relationship trees, the second metadata table taking the unique identifier of the first child node corresponding to the second child node and the unique identifier of the second child node as joint primary keys; constructing a second metadata description table based on the second metadata table, the second metadata description table taking the unique identifier of the second child node and the unique identifier of the first target value as joint primary keys.

6. The method of claim 1, wherein, Before obtaining each data source of each task for reading, the method further comprises: in response to a business collection request, collecting source data of a business system, the business collection request including a collection mode; after extracting and cleaning and converting the source data, obtaining target data; based on the collection mode, writing the target data into each data source involved in the business collection request; updating the target value corresponding to each data source.

7. A task scheduling apparatus characterized by comprising: The device comprises: a data source obtaining module, configured to obtain each data source of each task for reading; a first marker value obtaining module, configured to determine a first marker value of the task according to the target value corresponding to each data source, the target value representing the value of the data update of the data source at the last time; a second marker value obtaining module, configured to obtain a second marker value of the task, the second marker value being the first marker value determined by the task at the last execution; a data change determining module, configured to compare the first marker value and the second marker value to determine whether the data involved in the task has changed; a task triggering module, configured to trigger the execution of the task if the data involved in the task has changed. The first marker value obtaining module comprises: a coarse output mode obtaining unit, configured to determine the storage mode of the target value corresponding to each data source; a metadata table querying unit, configured to, if the storage mode of the target value is metadata table storage, obtain the target value corresponding to the data source by querying the metadata table corresponding to the data source; a non-metadata table querying unit, configured to, if the storage mode of the target value is non-metadata table storage, obtain the target value corresponding to the data source by querying the non-metadata table corresponding to the data source; a first marker value determining unit, configured to determine the maximum value among the target values corresponding to each data source obtained by querying as the first marker value.

8. The apparatus of claim 7, wherein, Each data source comprises at least one database, and the database comprises at least one business data table, and the first marker value obtaining module further comprises: a metadata table constructing unit, configured to construct the metadata table corresponding to the data source; the metadata table constructing unit comprises: a first constructing unit, configured to construct the dependency relationship tree corresponding to each database in the data source; a second constructing unit, configured to construct the metadata table corresponding to the data source according to each of the dependency relationship trees.

9. The apparatus of claim 8, wherein, The first constructing unit comprises: a root node constructing unit, configured to determine the root node of the dependency relationship tree according to the database; The first sub-node construction unit is configured to determine a first sub-node corresponding to each business data table in the database according to the each business data table, and the first sub-node is subordinate to the root node; The second sub-node construction unit is configured to, for any one business data table in the database, if the business data table is a partitioned business data table, determine a second sub-node corresponding to a partition in the business data table according to the partition, and the second sub-node is subordinate to the first sub-node corresponding to the business data table.

10. The apparatus of claim 9, wherein, The second construction unit comprises: The database metadata table construction unit is configured to construct a database metadata table according to all root nodes in each of the dependency relationship trees, and the database metadata table takes the unique identifier of the root node as a primary key; The first metadata table construction unit is configured to construct a first metadata table according to all first sub-nodes in each of the dependency relationship trees, and the first metadata table takes the unique identifier of the root node corresponding to the first sub-node and the unique identifier of the first sub-node as a joint primary key; The first metadata description table construction unit is configured to construct a first metadata description table based on the first metadata table, and the first metadata description table takes the unique identifier of the first sub-node and the unique identifier of a first target value as a joint primary key, and the first target value is a target value corresponding to the business data table corresponding to the first sub-node.

11. The apparatus of claim 10, wherein, The second construction unit further comprises: The second metadata table construction unit is configured to construct a second metadata table according to all second sub-nodes in each of the dependency relationship trees, and the second metadata table takes the unique identifier of the first sub-node corresponding to the second sub-node and the unique identifier of the second sub-node as a joint primary key; The second metadata description table construction unit is configured to construct a second metadata description table based on the second metadata table, and the second metadata description table takes the unique identifier of the second sub-node and the unique identifier of a first target value as a joint primary key.

12. The apparatus of claim 7, wherein, The apparatus further comprises: The data acquisition module is configured to acquire data from a business system, and update a database or a meta-database in a data warehouse according to the acquired data; The data acquisition module comprises: The source data acquisition unit is configured to acquire source data of the business system in response to a business acquisition request, and the business acquisition request comprises an acquisition mode; The source data processing unit is configured to obtain target data by extracting and cleaning and converting the source data; The data storage unit is configured to write the target data into each data source involved in the business acquisition request based on the acquisition mode; The metadata update unit is configured to update a target value corresponding to each data source.

13. A task scheduling device, characterized by The device comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to perform the task scheduling method of any one of claims 1-6.

14. A computer storage medium, characterized in that The computer storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the task scheduling method of any one of claims 1-6.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the electronic device to implement the task scheduling method as claimed in any one of claims 1-6.

Images