Data processing method, data processing apparatus, and computing device cluster

Abstract

Provided are a data processing method, a data processing apparatus, and a computing device cluster. The method comprises: acquiring first task information of a data processing platform, wherein the first task information comprises a plurality of job identifiers and upstream and downstream information corresponding to each of the plurality of job identifiers; on the basis of table lineage information of the data processing platform, updating upstream and downstream information corresponding to a first job identifier, wherein the first job identifier belongs to the plurality of job identifiers; updating the first task information on the basis of the updated upstream and downstream information corresponding to the first job identifier, so as to obtain second task information; and performing data processing on the basis of the second task information. By means of the method, unnecessary upstream and downstream relationships can be removed from first task information, thereby improving the efficiency of data processing.

Description

A data processing method, a data processing device, and a cluster of computing devices.

[0001] This application claims priority to Chinese Patent Application No. 202411442969.4, filed on October 15, 2024, entitled "A Data Processing Method, Data Processing Apparatus and Computing Device Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a data processing method, a data processing device, and a cluster of computing devices. Background Technology

[0003] In big data scenarios, updating data tables is a critical task. Generally, updating data usually requires multiple jobs. In other words, updating data involves the collaborative execution of multiple jobs.

[0004] In related technologies, task information is manually arranged, and then multiple jobs as instructed by the task information are executed. However, manually arranged task information can negatively impact data processing efficiency. Specifically, manually arranged task information may create unnecessary upstream and downstream relationships between jobs, which increases the consumption of additional computing resources, prolongs the data processing cycle, and thus reduces data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, a data processing device, and a computing device cluster, which can solve the problem of low data processing efficiency.

[0006] Firstly, this application provides a data processing method. This method can be applied to a data processing platform and includes: providing a user with first task information of the data processing platform, the first task information including multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers, the multiple job identifiers being used to identify each job, the upstream and / or downstream information being used to reflect the job execution order relationship between the various jobs, and each job being used to process one or more data tables on the data processing platform; obtaining a start job and an end job specified by the user, the start job and the end job being one or more of the various jobs, determining the correlation between the start job and the end job, removing irrelevant jobs between the start job and the end job to obtain second task information; and performing data processing based on the second task information.

[0007] In the above scheme, by determining the correlation between the user-specified start job and end job, and removing the irrelevant jobs between the start job and end job, the number of jobs that need to be executed can be reduced, and data processing efficiency can be improved.

[0008] In one possible implementation, the method further includes: obtaining the user-specified start data table and end data table; determining the start job based on the start data table; and determining the end job based on the end data table.

[0009] In one possible implementation, determining the correlation between the starting and ending jobs and removing irrelevant jobs between them to obtain the second task information specifically includes: identifying irrelevant jobs between the starting and ending jobs based on the table lineage information of the data tables processed by each job between the starting and ending jobs, and the upstream information of each job between the starting and ending jobs; removing the irrelevant jobs between the starting and ending jobs to obtain the second task information. Wherein, the upstream information corresponding to each job identifier among the plurality of job identifiers includes the upstream information of each job between the starting and ending jobs.

[0010] Taking the tasks between the starting task and the ending task, including the current task, as an example, the upstream tasks of the current task are determined based on its upstream information. If the data table processed by the current task has no table lineage relationship with the data tables processed by its upstream tasks, the current task is considered an unrelated task. Specifically, the table lineage information of the data tables processed by the current task includes a task list. This task list includes one or more task identifiers, and the data table processed by the task identifier in the task list is the parent data table of the data table processed by the current task. If the upstream information of the current task does not intersect with the task list, the current task is considered an unrelated task. It should be noted that if any task identifier in the upstream information of the current task is different from any task identifier in the task list, it means that the upstream information of the current task does not intersect with the task list. In some possible implementations, if the upstream information of the current task does not intersect with the task list, the current task and all its downstream tasks can also be considered unrelated tasks.

[0011] Taking the tasks between the starting task and the ending task, including the current task, as an example, if the data table processed by the current task has no table lineage relationship with the data table processed by an upstream task of the current task, the upstream relationship between the current task and that upstream task can be removed by updating the upstream information of the current task. The updated upstream information of the current task does not include the task identifier that identifies the upstream task.

[0012] In one possible implementation, the data processing based on the second task information includes: obtaining the refresh time specified by the user; and processing the data corresponding to the refresh time in the data table of the job processing identified by each job identifier in the second task information.

[0013] In one possible implementation, the data processing based on the second task information includes: obtaining the refresh time specified by the user; determining the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information based on the refresh time and the partition granularity of the data tables of the jobs identified by each job identifier in the second task information; and processing the data within the refresh time corresponding to the data tables of the jobs identified by each job identifier in the second task information according to the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information. The partition granularity of the data tables of the jobs identified by each job identifier in the second task information can be obtained from the attribute information of the jobs identified by each job identifier in the second task information. Furthermore, the partition granularity of the data tables of the jobs identified by each job identifier in the second task information can also be determined based on the format of the partition values ​​of the data tables of each job processing obtained from the data tables of each job processing.

[0014] In one possible implementation, the data processing based on the second task information includes: obtaining the refresh time and partition granularity specified by the user; determining the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information based on the refresh time and the partition granularity; and processing the data within the refresh time corresponding to the data table of the job processing identified by each job identifier in the second task information according to the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information.

[0015] In both of the above schemes, determining the number of instances according to the partition granularity of the data table can reduce duplicate instances and improve data processing efficiency.

[0016] In one possible implementation, the method further includes: obtaining the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; the processing of the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information specifically includes: processing the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information according to the scheduling frequency of the jobs identified by each job identifier in the second task information.

[0017] Secondly, this application also provides a data processing apparatus. The apparatus includes a providing module, an acquiring module, and a processing module.

[0018] The providing module is used to provide users with first task information of the data processing platform. The first task information includes multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information are used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform.

[0019] The acquisition module is further configured to acquire the user-specified start job and end job, wherein the start job and end job are one or more jobs among the various jobs, determine the correlation between the start job and end job, and remove irrelevant jobs between the start job and end job to obtain second task information.

[0020] The processing module is used to process data based on the second task information.

[0021] In one possible implementation, the acquisition module is further configured to: acquire the user-specified start data table and end data table; determine the start job based on the start data table; and determine the end job based on the end data table.

[0022] In one possible implementation, the acquisition module is further configured to: determine unrelated jobs between the starting job and the ending job based on the table lineage information of the data tables of each job processing between the starting job and the ending job, and the upstream information of each job between the starting job and the ending job; remove the unrelated jobs between the starting job and the ending job to obtain the second task information. Wherein, the upstream information corresponding to each job identifier in the plurality of job identifiers includes the upstream information of each job between the starting job and the ending job.

[0023] In one possible implementation, the processing module is specifically used to: obtain the refresh time specified by the user; and process the data corresponding to the refresh time in the data table of each job identifier in the second task information.

[0024] In one possible implementation, the processing module is specifically configured to: obtain the refresh time specified by the user; determine the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information based on the refresh time and the partition granularity of the data tables of the jobs identified by each job identifier in the second task information; and process the data within the refresh time corresponding to the data tables of the jobs identified by each job identifier in the second task information according to the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information.

[0025] In one possible implementation, the processing module is specifically used to: obtain the refresh time and partition granularity specified by the user; determine the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information according to the refresh time and the partition granularity; and process the data within the refresh time corresponding to the data table of the job processing identified by each job identifier in the second task information according to the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information.

[0026] In one possible implementation, the processing module is further configured to: obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; the processing of the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information specifically includes: processing the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information according to the scheduling frequency of the jobs identified by each job identifier in the second task information.

[0027] Thirdly, this application also provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the data processing method provided by the first aspect or any possible implementation thereof.

[0028] Fourthly, this application also provides a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the data processing method provided by the first aspect or any possible implementation thereof.

[0029] Fifthly, this application also provides a computer program product, including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the data processing method provided by the first aspect or any possible implementation thereof.

[0030] The aforementioned apparatus, computing device cluster, computer storage medium, or computer program product are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding solutions in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0031] Figure 1 is a schematic diagram of task information provided in an embodiment of this application;

[0032] Figure 2 is a flowchart of a data processing method provided in an embodiment of this application;

[0033] Figures 3a and 3b are schematic diagrams of a configuration supplementary data information provided in an embodiment of this application;

[0034] Figures 4 and 5 are schematic diagrams of an updated task information provided in an embodiment of this application;

[0035] Figure 6 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0036] Figure 7 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0037] Figure 8 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0038] Figure 9 is a schematic diagram of a computing device cluster in which the device shown in Figure 6 is deployed, according to an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0040] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0041] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0043] Before introducing the embodiments of this application, the terms mentioned in the embodiments of this application will be explained below.

[0044] Table lineage information describes the dependencies between data tables. A dependency is formed when data in one data table (called the destination table) originates from or is calculated from data in one or more other data tables (called source tables). The source table can be called the parent table of the destination table, and the destination table can be called the child table of the source tables. It's important to note that the terms "parent table" and "child table" are relative; a data table can be both the parent and child table of another data table.

[0045] For example, in big data scenarios, the initial data table undergoes different levels of processing to generate different levels of data tables. Each level of processing adds new features to the previous level's data table, thus generating a new data table. In other words, there is a lineage between data tables at different levels. For instance, the first level of processing might involve adding first-level features to the initial data table to generate a first-level data table, the second level of processing might involve adding second-level features to generate a new data table, and so on. The initial data table can include a data table obtained from the source database. The first-level data table refers to the new data table obtained after the first level of processing; the meanings of other levels of data tables are similar and will not be elaborated here. In some embodiments, the data tables at each level in a big data scenario may include, but are not limited to, data tables at the operational data store (ODS), data warehouse detail (DWD), data warehouse summary (DWS), and analytical data store (ADS) levels. In some embodiments, an ODS layer data table can be generated based on the source data table, then a DWD layer data table can be calculated based on the ODS layer data table, then a DWS layer data table can be calculated based on the DWD layer data table, and finally an ADS data table can be calculated based on the DWD layer or DWS layer data table. Here, the ODS layer data table refers to a new data table generated by overlaying features from the ODS layer. The meanings of the DWD layer data table, DWS layer data table, and ADS data table are similar and will not be elaborated further here.

[0046] Job lineage information describes the execution order relationship between jobs, i.e., upstream and downstream relationships. A job is the carrier of the computational logic of a data table. Specifically, a job can be a program used to process data in a data table, and the specific processing procedures can include, but are not limited to, operations such as updating, deleting, and inserting. In practical applications, when the processing of multiple data tables cannot be achieved by a single job, the processing of these multiple data tables can be divided into different jobs according to certain partitioning rules, and these processing procedures can be executed by different jobs. After the processing of multiple data tables is divided, upstream and downstream relationships will exist between the jobs. The job lineage information of a job can specifically include upstream information and / or downstream information. Upstream information includes at least the job identifier that identifies the upstream job of this job, and downstream information includes at least the job identifier that identifies the downstream job of this job. For example, in big data scenarios, partitioning rules can include, but are not limited to, one or more of the following factors: data table generation hierarchy, data table lineage information, business type, and priority. When there is an upstream and downstream relationship between two jobs, the upstream job can also be called the parent job, and the downstream job can also be called the child job. It is worth noting that the terms upstream job and downstream job (and parent job and child job) are relative. A job can act as both an upstream job (parent job) and a downstream job (child job) of another job. The job identifier that identifies an upstream job can also be called the parent job identifier, and the job identifier that identifies an upstream job can also be called the child job identifier.

[0047] Data replenishment (or data refresh / update) jobs refer to operations that update data in big data scenarios under certain circumstances. For example, when business rules change or data errors occur in a big data scenario, data replenishment jobs can be used to update data tables. Data tables in big data scenarios can be divided into multiple partitions based on time. The partition granularity of a data table can be daily, monthly, or yearly. Different data tables can have different partition granularities. For data tables partitioned by time, daily routine big data jobs can process data from the most recent partition. For example, a big data job for a data table T with a specific daily partition might be executed at 3 AM daily, processing data from the most recent three days each time it's executed. When a problem is found in the data for a specific time period X in data table T, a data replenishment job can be used to update the data for that time period X. In other words, after executing this data replenishment job, the data for time period X in data table T will be updated.

[0048] The partition field in a data table is used to indicate the granularity of partitioning. In big data scenarios, the partition granularity of a data table can be determined based on the value of the partition field, i.e., whether it is a daily, monthly, or yearly partition. For example, if the partition field value is in the format yyyy-mm-dd (yyyy represents year, mm represents month, and dd represents day), the partition granularity of the data table can be determined as a daily partition. Similarly, if the partition field value is in the format yyyy-mm, the partition granularity can be determined as a monthly partition. Finally, if the partition field value is in the format yyyy, the partition granularity can be determined as a yearly partition.

[0049] A directed acyclic graph (DAG) is a visual representation of task information. Task information describes the execution order of various jobs. Each node in a job DAG represents a job, and directed edges between nodes represent the execution order relationship between jobs. That is, if a directed edge exists between two nodes in a job DAG, it indicates that there is an execution order relationship between the jobs represented by those two nodes. If there is an execution order relationship between two jobs, the job executed earlier can be called the upstream job of the job executed later, and the job executed later can be called the downstream job of the job executed earlier. The directionality of a job DAG ensures that the execution order of jobs is explicit. The acyclicity of a job DAG ensures that there are no loops in the execution process, avoiding problems such as deadlock.

[0050] In big data scenarios, due to the existence of table lineages, updating one table also requires updating its child tables. This updating process necessitates the creation of jobs that process these tables. Executing some jobs also triggers downstream jobs. In other words, when data needs to be supplemented in a big data scenario, a large number of jobs are executed. To effectively manage and execute these jobs, relevant technologies utilize job directive AG orchestration and define the execution order relationships between jobs.

[0051] Figure 1 is a visualization diagram of task information in a big data scenario provided by an embodiment of this application. As shown in Figure 1, the task information is represented by a job DAG. The jobs represented by the nodes in the job DAG include jobs A1-A2, jobs B1-B2, jobs C1-C3, and jobs D1-D3.

[0052] Figure 1 shows arrowed line segments connecting tasks, indicating the execution order between the two tasks. For example, task A1 and task A2 are connected by an arrowed line segment, indicating that task A1 is executed first, followed by task A2. Task A1 can be considered the upstream task of task A2, and task A2 can be considered the downstream task of task A1. The meanings of other tasks and their connecting arrowed line segments in Figure 1 are similar and will not be repeated here.

[0053] However, the arrangement of the job DAG shown in Figure 1 involves many factors, which may add unnecessary upstream and downstream relationships between jobs in the job DAG. This not only consumes valuable resources (such as computing and storage resources) but also increases the time of the entire data processing flow, thus leading to a decrease in data processing efficiency.

[0054] Therefore, this application provides a data processing method that can solve the above problems.

[0055] In the method provided in this application embodiment, after providing the user with the first task information of the data processing platform, the method obtains the start and end jobs specified by the user based on the first task information, determines the correlation between the start and end jobs specified by the user, removes irrelevant jobs between the start and end jobs, obtains the second task information, and performs data processing based on the second task information. This method can reduce the number of jobs that need to be executed, reduce resource consumption, and avoid increasing data processing time, thereby improving data processing efficiency.

[0056] The data processing method provided in the embodiments of this application will be described in detail below with reference to Figure 2.

[0057] Figure 2 is a flowchart of a data processing method provided in an embodiment of this application. The method shown in Figure 2 can be applied to a data processing platform in a big data scenario, and can be executed by one computing device in the data processing platform, or by multiple computing devices working together.

[0058] As shown in Figure 2, the method may include steps S201 to S204. The following description, using a single computing device as an example, refers to Figure 1 to illustrate the steps included in the method shown in Figure 2. When multiple computing devices collaboratively execute the steps included in the method shown in Figure 2, each computing device may execute one or more steps included in the method shown in Figure 2.

[0059] S201 provides users with the first task information of the data processing platform.

[0060] The computing device of the data processing platform can obtain pre-stored first task information from the storage device in the data processing platform or from the storage space of the computing device, and then provide the first task information in a visual manner through the display interface, or provide the first task information to the user through the application programming interface.

[0061] The first task information is structured data. Specifically, the first task information may include multiple job identifiers, as well as upstream and / or downstream information corresponding to each job identifier. The multiple job identifiers are used to identify each job. The upstream and / or downstream information corresponding to each job identifier reflects the job execution order relationship between the jobs. Each job is used to process one or more data tables on the data processing platform.

[0062] Taking the job DAG shown in Figure 1 as an example, the first task information includes multiple job identifiers, which can be the identifiers of jobs A1 to A2, jobs B1 to B2, jobs C1 to C3, and jobs D1 to D3.

[0063] In this context, tasks A1 and A2 are the starting tasks. There are no upstream tasks, but there are downstream tasks. That is, the identifiers of tasks A1 and A2 have corresponding downstream information, but no corresponding upstream information (or the upstream information is empty). Taking task A1 as an example, the downstream information of task A1 can include the identifier of task B1.

[0064] Among them, tasks B1-B2 and C1-C3 are intermediate tasks, with upstream and downstream components. That is, the identifiers of tasks B1-B2 and C1-C3 have corresponding upstream and downstream information. Taking task B1 as an example, its upstream information can include the identifier of task A1, and its downstream information can include the identifiers of tasks C1 and C2. Taking task C2 as an example, its upstream information can include the identifiers of tasks B1 and B2, and its downstream information can include the identifier of task D2.

[0065] Among them, tasks D1 to D3 are completed tasks, with upstream tasks but no downstream tasks. That is, the identifiers of tasks D1 to D3 have corresponding upstream information, but no corresponding downstream information (or the downstream information is empty). Taking task D3 as an example, the upstream information of task D3 can include the identifier of task C3.

[0066] The first task information can be generated based on the job lineage information of the data processing platform. In this case, the upstream job of a job identified by a job identifier is also called the parent job of that job, and the downstream job of that job is also called the child job of that job.

[0067] S202, obtain the user-specified start job and end job, where the start job and end job are one or more jobs identified by multiple job identifiers in the first task information.

[0068] In big data scenarios, users can provide supplementary data information to the computing devices of a data processing platform through their terminal devices. The user's terminal device can have a client application for the data processing platform deployed on it. When a user needs supplementary data, they can generate the supplementary data information through this client and then send it to the data processing platform through their terminal device. Specifically, the user can run the client on their terminal device, access the supplementary data page through the client, and configure the supplementary data information there.

[0069] Specifically, supplementary data information may include start information and end information. The start information may include the start data table identifier and / or the start job identifier. The end information may include one or more of the following: end data table identifier, end job identifier, and selection of all downstream tasks. It is understood that the supplementary data information may include one or more identifiers such as the start data table identifier, start job identifier, end data table identifier, and end job identifier. As shown in Figure 3a, after entering the supplementary data page, the user can select the start data table, select the start job, select the end data table, select the end job, and select all downstream jobs of the start job.

[0070] In some embodiments, the supplementary data information may further include the values ​​of functional parameters. These functional parameters indicate whether irrelevant jobs are removed. As shown in Figure 3b, after entering the supplementary data page, the user can select the starting data table, the starting job, the ending data table, the ending job, and all downstream jobs of the starting job. The user can also configure the values ​​of the functional parameters. When the user-configured functional parameter value indicates that irrelevant jobs should not be removed, the computing device may not execute S202 and S203.

[0071] The computing device can determine the start and end jobs specified by the user based on the received supplementary data information.

[0072] Taking a scenario where the starting information includes a starting job identifier and the ending information includes an ending job identifier as an example, the computing device can query the first task information of the data processing platform based on the starting job identifier and the ending job identifier to obtain the job between the starting job and the ending job. For example: querying the first task information based on the starting job identifier yields the upstream and downstream information corresponding to the starting job identifier. Then, the device continues to query the first task information based on the job identifiers contained in the upstream and downstream information corresponding to the starting job identifier. The query ends when the job identifiers contained in the continuously queried upstream and downstream information are the same as the ending job identifier.

[0073] Taking an example where the starting information includes a starting data table identifier and the ending information includes an ending data table identifier, the computing device can use the job that processes the data table identified by the starting data table identifier as the starting job and the job that processes the data table identified by the ending data table identifier as the ending job. Then, based on the starting job identifier and the ending job identifier, it queries the first task information to obtain the jobs between the starting job and the ending job. The specific process by which the computing device determines the jobs between the starting job and the ending job can be referred to the explanation in the previous paragraph, and will not be repeated here. The specific process by which the computing device determines the starting job and the ending job may include: the computing device can query the table lineage information of the starting data table based on the starting data table identifier to obtain the starting job identifier; similarly, it can query the table lineage information of the ending data table based on the ending data table identifier to obtain the ending job identifier.

[0074] Taking the starting information as including the starting job identifier and the ending information as including the selection of all downstream task information as an example, the computing device can query the first task information based on the starting job identifier to obtain the upstream and downstream information corresponding to the starting job identifier. Then, it can continue to query the first task information based on the upstream and downstream information corresponding to the starting job identifier until the upstream and downstream information of the data processing platform has been queried.

[0075] S203, determine the correlation between the starting task and the ending task, remove the irrelevant tasks between the starting task and the ending task to obtain the second task information.

[0076] Specifically, the computing device can traverse all jobs between the starting job and the ending job, and based on the table lineage information of the data tables processed by each job between the starting job and the ending job, as well as the upstream information of each job between the starting job and the ending job, determine the unrelated jobs between the starting job and the ending job, remove the unrelated jobs between the starting job and the ending job, and obtain the second task information. The upstream information corresponding to each of the multiple job identifiers includes the upstream information of each job between the starting job and the ending job.

[0077] The process by which a computing device determines unrelated jobs between a starting job and an ending job may include: taking the current job as an example, among the jobs between the starting and ending jobs, the upstream jobs of the current job are determined based on the upstream information of the current job. If the data table processed by the current job has no table lineage relationship with the data tables processed by the upstream jobs of the current job, the current job is considered an unrelated job. Specifically, the table lineage information of the data table processed by the current job includes a job list, which includes one or more job identifiers, and the data table processed by the job identifier in the job list is the parent data table of the data table processed by the current job. If any job identifier in the upstream information of the current job is different from the job identifier in the job list, it indicates that the upstream information of the current job and the job list have no intersection, that is, the data table processed by the current job has no table lineage relationship with the data tables processed by the upstream jobs of the current job, and the current job is then considered an unrelated job. In some possible implementations, the current job and all its downstream jobs may also be considered unrelated jobs.

[0078] For example, taking job A1 as the starting job in Figure 1, the downstream jobs of job A1 include job B1, job C2, job D2, job C1, and job D1. Taking job B1 as the current job, if the job lists included in the table lineage information of the data tables processed by job A1 and job B1 do not overlap, job B1 can be considered an unrelated job. Taking job C2 as the current job, if the job lists included in the table lineage information of the data tables processed by job B1 and job B2 do not overlap, job C2 can be considered an unrelated job.

[0079] The computing device removes irrelevant jobs between the starting and ending jobs to obtain second task information. Specifically, this may include removing the job identifiers and their corresponding upstream / downstream information of irrelevant jobs from the first task information. Taking all downstream jobs of the starting job, including the current job, as an example, the job identifier of the current job and its corresponding upstream / downstream information can be removed from the first task information.

[0080] Furthermore, if the data table processed by the current job has no table lineage relationship with the data table processed by an upstream job, the job identifier of that upstream job can be removed from the upstream information of the current job. This removes the upstream-downstream relationship between the current job and that upstream job, resulting in the second task information. For example, for job C2, when the job identifiers in the job lists included in the table lineage information of the data tables processed by job B1 and job C2 are all different, as shown in Figure 4, the job identifier of job B1 in the upstream information of job C2 can be removed, thereby removing the upstream-downstream relationship between job B1 and job C2. In other words, the updated upstream information corresponding to job C2 does not include the job identifier of job B1.

[0081] Furthermore, when the upstream information corresponding to the updated current job is empty, both the current job and its downstream jobs can be removed. Taking job C2 as the current job in Figure 1 as an example, as shown in Figure 5, if the upstream and downstream relationships between jobs B1 and B2 and job C2 are removed, job C2 and its downstream job D2 can be removed. Removing job C2 and its downstream job D2 can include removing job C2 and its downstream information, as well as job D2 and its upstream information, from the first task information.

[0082] S204, perform data processing based on the second task information.

[0083] After obtaining the second task information, the computing device can execute each task sequentially according to the task identifier and its corresponding upstream and downstream information included in the second task information.

[0084] Users can also configure the refresh time on the data replenishment page. In this case, the computing device can process data according to the user-specified refresh time.

[0085] Specifically, the computing device can obtain the refresh time specified by the user, and determine the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information based on the refresh time and the partition granularity of the data tables of the jobs identified by each job identifier in the second task information; and process the data within the refresh time corresponding to the data tables of the jobs identified by each job identifier in the second task information according to the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information.

[0086] Specifically, the computing device can also obtain the refresh time and partition granularity specified by the user; based on the refresh time and the partition granularity, determine the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information; and process the data within the refresh time corresponding to the data table of the job processing identified by each job identifier in the second task information according to the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information.

[0087] Specifically, the computing device can also obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; the processing of the data in the data table of the jobs identified by each job identifier in the second task information within the refresh time specifically includes: processing the data in the data table of the jobs identified by each job identifier in the second task information within the refresh time according to the scheduling frequency of the jobs identified by each job identifier in the second task information.

[0088] In big data scenarios, jobs are typically executed daily, meaning the execution cycle is one day. However, the partitioning granularity of the data tables processed by these jobs can be daily, monthly, or yearly. In related technologies, when data needs to be supplemented, daily jobs generate a new instance each day based on the time range to be updated. For monthly and yearly partitioned tables, this results in a large amount of redundant computation and resource consumption. Therefore, computing devices can determine the number of instances N based on the partitioning granularity of the data table to be processed by the job, and then process the data in the table using N instances. Determining the number of instances according to the partitioning granularity of the data table reduces redundant instances and further improves data processing efficiency.

[0089] Specifically, the computing device can determine the number of instances corresponding to the data table of each job identified by the job identifier in the second task information, which may include: when the partition granularity of the data table is daily, the number of instances N is determined by the refresh time per day; when the partition granularity of the data table is monthly, the number of instances N is determined by the refresh time per month; when the partition granularity of the data table is yearly, the number of instances N is determined by the refresh time per year. Taking a refresh time of 01 / 01-02 / 29 of a certain year as an example, when the partition granularity of the data table is daily, the number of instances N is 60; when the partition granularity of the data table is monthly, the number of instances N is 2; and when the partition granularity of the data table is yearly, the number of instances N is 1.

[0090] Specifically, the partition granularity of the data table for each job identified by each job identifier in the second task information can be obtained from the attribute information of the job identified by each job identifier in the second task information. Furthermore, the partition granularity of the data table for each job identified by each job identifier in the second task information can also be determined based on the format of the partition values ​​of the data tables for each job processing obtained from the data tables of each job processing.

[0091] In the method embodiment shown in Figure 2 above, by determining the correlation between the user-specified start job and end job, and removing the irrelevant jobs between the start job and the end job, the number of jobs that need to be executed can be reduced, thereby improving data processing efficiency.

[0092] Based on the method embodiment shown in FIG2, this application embodiment also provides a data processing apparatus. This data processing apparatus can be used to perform the various steps in the method shown in FIG2.

[0093] Figure 6 is a schematic diagram of the structure of a data processing device 600 provided in an embodiment of this application. As shown in Figure 6, the data processing device 600 may include a providing module 601, an acquiring module 602, and a processing module 603. When the data processing device 600 is used to execute the various steps in the method shown in Figure 2 above, the functions of each module are as described below.

[0094] The providing module 601 is used to provide the user with first task information of the data processing platform. The first task information includes multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information are used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform.

[0095] The acquisition module 602 is used to acquire the start job and end job specified by the user, wherein the start job and end job are one or more jobs among the various jobs, determine the correlation between the start job and end job, and remove irrelevant jobs between the start job and end job to obtain the second task information.

[0096] The processing module 603 is used to perform data processing based on the second task information.

[0097] It should be noted that the data processing device 600 provided in the embodiment shown in Figure 6 is only illustrated by the division of the above-mentioned functional modules when executing the data processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the computing device provided in the above embodiment and the data processing method embodiment shown in Figure 2 belong to the same concept. For details of its specific implementation process, please refer to the method embodiment and the invention content, which will not be repeated here.

[0098] When the above-described module is used as an example of a software functional unit, the data processing device 600 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Further, the aforementioned computing device may be one or more. For example, the data processing device 600 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0099] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0100] When the above-mentioned module is used as an example of a hardware functional unit, the module may include at least one computing device, such as a server. Alternatively, the module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0101] The multiple computing devices included in a module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in a module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in a module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0102] It should be noted that, in other embodiments, the providing module 601, the acquiring module 602, and the processing module 603 can be used to execute any step in the method shown in FIG2. The steps implemented by the providing module 601, the acquiring module 602, and the processing module 603 can be specified as needed. By implementing different steps in the method shown in FIG2 through the providing module 601, the acquiring module 602, and the processing module 603, all functions of the data processing device 600 can be realized.

[0103] This application also provides a computing device 700. As shown in FIG7, the computing device 700 includes: a processor 701, a memory 702, a communication interface 703, and a bus 704. The processor 701, the memory 702, and the communication interface 703 communicate with each other via the bus 704. The computing device 700 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0104] Bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 7, but this does not imply that there is only one bus or one type of bus. Bus 104 can include pathways for transmitting information between various components of computing device 700 (e.g., memory 702, processor 701, communication interface 703).

[0105] Processor 701 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0106] The memory 702 may include volatile memory, such as random access memory (RAM). The processor 701 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0107] The memory 702 stores executable program code, and the processor 701 executes this executable program code to implement the functions of the aforementioned providing module 601, obtaining module 602, and processing module 603, thereby implementing the method shown in FIG2. That is, the memory 702 stores instructions for executing the method shown in FIG2. And / or, the memory 702 stores executable code, and the processor 701 executes this executable code to implement the functions of the aforementioned providing module 601, obtaining module 602, and processing module 603, thereby implementing the method shown in FIG7. That is, the memory 702 stores instructions for executing the method shown in FIG7.

[0108] The communication interface 703 uses modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 700 and other devices or communication networks.

[0109] Based on the method embodiment shown in Figure 2, this application embodiment also provides a computing device cluster.

[0110] Figure 8 shows a computing device cluster 800 provided in an embodiment of this application.

[0111] The computing device cluster 800 may include the data processing platform described above. The computing device cluster includes at least one computing device. This computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0112] As shown in Figure 8, the computing device cluster includes at least one computing device 700. The memory 702 of one or more computing devices 700 in the computing device cluster may store the same instructions for executing the method shown in Figure 2.

[0113] In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the method shown in FIG2. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the method shown in FIG2.

[0114] It should be noted that the memories 702 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device 600. That is, the instructions stored in the memories 702 of different computing devices 700 can implement the functions of one or more modules among the providing module 601, the acquiring module 602, and the processing module 603.

[0115] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 illustrates one possible implementation. As shown in Figure 9, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 702 in computing device 700A stores instructions for performing the functions of the providing module 601 and the obtaining module 602. Simultaneously, the memory 702 in computing device 700B stores instructions for performing the functions of the processing module 603. The connection method between the computing device clusters shown in Figure 9 can be considered because the data processing method provided in this application requires a large amount of storage and computing resources (e.g., storing large amounts of data), therefore, the functions implemented by the processing module 603 are considered to be performed by computing device 700B. It should be understood that the functions of computing device 700A shown in Figure 9 can also be performed by multiple computing devices 700. Similarly, the functions of computing device 700B can also be performed by multiple computing devices 700.

[0116] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 8 and 9. The difference is that the memory 702 of one or more computing devices 700 in this computing device cluster can store the same instructions for executing the data processing method shown in Figure 2.

[0117] In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the data processing method shown in FIG2. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the data processing method shown in FIG2.

[0118] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the data processing method shown in FIG2.

[0119] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the data processing method shown in FIG2, or instruct the computing device to perform the data processing method shown in FIG2.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, Applied to a data processing platform, the method includes: The data processing platform provides users with first task information, which includes multiple job identifiers and upstream and / or downstream information corresponding to each job identifier. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information is used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform. Obtain the user-specified start job and end job, wherein the start job and end job are one or more of the various jobs; Determine the correlation between the starting task and the ending task, and remove irrelevant tasks between the starting task and the ending task to obtain the second task information; Data processing is performed based on the second task information.

2. The method according to claim 1, characterized in that, The method further includes: Retrieve the user-specified start and end data tables; The starting job is determined based on the starting data table, and the ending job is determined based on the ending data table.

3. The method according to claim 1 or 2, characterized in that, The step of determining the correlation between the starting job and the ending job, and removing irrelevant jobs between the starting job and the ending job to obtain the second task information specifically includes: Based on the table lineage information of the data tables of each job processing between the starting job and the ending job, and the upstream information of each job between the starting job and the ending job, unrelated jobs between the starting job and the ending job are determined. The upstream information corresponding to each job identifier in the plurality of job identifiers includes the upstream information of each job between the starting job and the ending job. Remove irrelevant tasks between the start and end tasks to obtain the second task information.

4. The method according to any one of claims 1-3, characterized in that, The data processing based on the second task information includes: Get the refresh time specified by the user; The data corresponding to the refresh time in the data table of each job identifier in the second task information is processed.

5. The method according to any one of claims 1-3, characterized in that, The data processing based on the second task information includes: Get the refresh time specified by the user; Based on the refresh time and the partition granularity of the data table processed by each job identifier in the second task information, determine the number of instances corresponding to the data table processed by each job identifier in the second task information. Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.

6. The method according to any one of claims 1-3, characterized in that, The data processing based on the second task information includes: Obtain the refresh time and partition granularity specified by the user; Based on the refresh time and the partition granularity, determine the number of instances corresponding to the data table processed by each job identifier in the second task information; Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.

7. The method according to any one of claims 4-6, characterized in that, The method further includes: Obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; The processing of the data within the refresh time corresponding to each job identifier in the data table of the second task information specifically includes: According to the scheduling frequency of the jobs identified by each job identifier in the second task information, the data within the refresh time corresponding to the data table of the jobs processed by each job identifier in the second task information is processed.

8. A data processing apparatus, characterized in that, The device, applied to a data processing platform, includes: A providing module is used to provide users with first task information of the data processing platform. The first task information includes multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information are used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform. The acquisition module is further configured to acquire the user-specified start job and end job, wherein the start job and end job are one or more jobs among the various jobs, determine the correlation between the start job and end job, and remove irrelevant jobs between the start job and end job to obtain second task information; The processing module is used to process data based on the second task information.

9. The apparatus according to claim 8, characterized in that, The acquisition module is also used for: Retrieve the user-specified start and end data tables; The starting job is determined based on the starting data table, and the ending job is determined based on the ending data table.

10. The apparatus according to claim 8 or 9, characterized in that, The acquisition module is also used for: Based on the table lineage information of the data tables of each job processing between the starting job and the ending job, and the upstream information of each job between the starting job and the ending job, unrelated jobs between the starting job and the ending job are determined. The upstream information corresponding to each job identifier in the plurality of job identifiers includes the upstream information of each job between the starting job and the ending job. Remove irrelevant tasks between the start and end tasks to obtain the second task information.

11. The apparatus according to any one of claims 8-10, characterized in that, The processing module is specifically used for: Get the refresh time specified by the user; The data corresponding to the refresh time in the data table of each job identifier in the second task information is processed.

12. The apparatus according to any one of claims 8-10, characterized in that, The processing module is specifically used for: Get the refresh time specified by the user; Based on the refresh time and the partition granularity of the data table processed by each job identifier in the second task information, determine the number of instances corresponding to the data table processed by each job identifier in the second task information. Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.

13. The apparatus according to any one of claims 8-10, characterized in that, The processing module is specifically used for: Obtain the refresh time and partition granularity specified by the user; Based on the refresh time and the partition granularity, determine the number of instances corresponding to the data table processed by each job identifier in the second task information; Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.

14. The apparatus according to any one of claims 11-13, characterized in that, The processing module is also used for: Obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; The processing of the data within the refresh time corresponding to each job identifier in the data table of the second task information specifically includes: According to the scheduling frequency of the jobs identified by each job identifier in the second task information, the data within the refresh time corresponding to the data table of the jobs processed by each job identifier in the second task information is processed.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, Includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

17. A computer program product, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-7.

Images