Data processing method, data processing device and computing device cluster
By determining the correlation between the starting and ending jobs in big data scenarios, irrelevant jobs are removed, and second task information is generated, thus solving the problem of resource waste caused by manual scheduling and improving data processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
- Filing Date
- 2024-10-15
- Publication Date
- 2026-04-17
Smart Images
Figure CN121880322A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data processing method, a data processing device, and a cluster of computing devices. Background Technology
[0002] In big data scenarios, updating data tables is a critical task. Generally, updating data usually requires multiple jobs. In other words, updating data involves the collaborative execution of multiple jobs.
[0003] In related technologies, task information is manually arranged, and then multiple jobs as instructed by the task information are executed. However, manually arranged task information can negatively impact data processing efficiency. Specifically, manually arranged task information may create unnecessary upstream and downstream relationships between jobs, which increases the consumption of additional computing resources, prolongs the data processing cycle, and thus reduces data processing efficiency. Summary of the Invention
[0004] This application provides a data processing method, a data processing device, and a computing device cluster, which can solve the problem of low data processing efficiency.
[0005] Firstly, this application provides a data processing method. This method can be applied to a data processing platform and includes: providing a user with first task information of the data processing platform, the first task information including multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers, the multiple job identifiers being used to identify each job, the upstream and / or downstream information being used to reflect the job execution order relationship between the various jobs, and each job being used to process one or more data tables on the data processing platform; obtaining a start job and an end job specified by the user, the start job and the end job being one or more of the various jobs, determining the correlation between the start job and the end job, removing irrelevant jobs between the start job and the end job to obtain second task information; and performing data processing based on the second task information.
[0006] In the above scheme, by determining the correlation between the user-specified start job and end job, and removing the irrelevant jobs between the start job and end job, the number of jobs that need to be executed can be reduced, and data processing efficiency can be improved.
[0007] In one possible implementation, the method further includes: obtaining the user-specified start data table and end data table; determining the start job based on the start data table; and determining the end job based on the end data table.
[0008] In one possible implementation, determining the correlation between the starting and ending jobs and removing irrelevant jobs between them to obtain the second task information specifically includes: identifying irrelevant jobs between the starting and ending jobs based on the table lineage information of the data tables processed by each job between the starting and ending jobs, and the upstream information of each job between the starting and ending jobs; removing the irrelevant jobs between the starting and ending jobs to obtain the second task information. Wherein, the upstream information corresponding to each job identifier among the plurality of job identifiers includes the upstream information of each job between the starting and ending jobs.
[0009] Taking the tasks between the starting task and the ending task, including the current task, as an example, the upstream tasks of the current task are determined based on its upstream information. If the data table processed by the current task has no table lineage relationship with the data tables processed by its upstream tasks, the current task is considered an unrelated task. Specifically, the table lineage information of the data tables processed by the current task includes a task list. This task list includes one or more task identifiers, and the data table processed by the task identifier in the task list is the parent data table of the data table processed by the current task. If the upstream information of the current task does not intersect with the task list, the current task is considered an unrelated task. It should be noted that if any task identifier in the upstream information of the current task is different from any task identifier in the task list, it means that the upstream information of the current task does not intersect with the task list. In some possible implementations, if the upstream information of the current task does not intersect with the task list, the current task and all its downstream tasks can also be considered unrelated tasks.
[0010] Taking the tasks between the starting task and the ending task, including the current task, as an example, if the data table processed by the current task has no table lineage relationship with the data table processed by an upstream task of the current task, the upstream relationship between the current task and that upstream task can be removed by updating the upstream information of the current task. The updated upstream information of the current task does not include the task identifier that identifies the upstream task.
[0011] In one possible implementation, the data processing based on the second task information includes: obtaining the refresh time specified by the user; and processing the data corresponding to the refresh time in the data table of the job processing identified by each job identifier in the second task information.
[0012] In one possible implementation, the data processing based on the second task information includes: obtaining the refresh time specified by the user; determining the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information based on the refresh time and the partition granularity of the data tables of the jobs identified by each job identifier in the second task information; and processing the data within the refresh time corresponding to the data tables of the jobs identified by each job identifier in the second task information according to the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information. The partition granularity of the data tables of the jobs identified by each job identifier in the second task information can be obtained from the attribute information of the jobs identified by each job identifier in the second task information. Furthermore, the partition granularity of the data tables of the jobs identified by each job identifier in the second task information can also be determined based on the format of the partition values of the data tables of each job processing obtained from the data tables of each job processing.
[0013] In one possible implementation, the data processing based on the second task information includes: obtaining the refresh time and partition granularity specified by the user; determining the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information based on the refresh time and the partition granularity; and processing the data within the refresh time corresponding to the data table of the job processing identified by each job identifier in the second task information according to the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information.
[0014] In both of the above schemes, determining the number of instances according to the partition granularity of the data table can reduce duplicate instances and improve data processing efficiency.
[0015] In one possible implementation, the method further includes: obtaining the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; the processing of the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information specifically includes: processing the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information according to the scheduling frequency of the jobs identified by each job identifier in the second task information.
[0016] Secondly, this application also provides a data processing apparatus. The apparatus includes a providing module, an acquiring module, and a processing module.
[0017] The providing module is used to provide users with first task information of the data processing platform. The first task information includes multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information are used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform.
[0018] The acquisition module is further configured to acquire the user-specified start job and end job, wherein the start job and end job are one or more jobs among the various jobs, determine the correlation between the start job and end job, and remove irrelevant jobs between the start job and end job to obtain second task information.
[0019] The processing module is used to process data based on the second task information.
[0020] In one possible implementation, the acquisition module is further configured to: acquire the user-specified start data table and end data table; determine the start job based on the start data table; and determine the end job based on the end data table.
[0021] In one possible implementation, the acquisition module is further configured to: determine unrelated jobs between the starting job and the ending job based on the table lineage information of the data tables of each job processing between the starting job and the ending job, and the upstream information of each job between the starting job and the ending job; remove the unrelated jobs between the starting job and the ending job to obtain the second task information. Wherein, the upstream information corresponding to each job identifier in the plurality of job identifiers includes the upstream information of each job between the starting job and the ending job.
[0022] In one possible implementation, the processing module is specifically used to: obtain the refresh time specified by the user; and process the data corresponding to the refresh time in the data table of each job identifier in the second task information.
[0023] In one possible implementation, the processing module is specifically configured to: obtain the refresh time specified by the user; determine the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information based on the refresh time and the partition granularity of the data tables of the jobs identified by each job identifier in the second task information; and process the data within the refresh time corresponding to the data tables of the jobs identified by each job identifier in the second task information according to the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information.
[0024] In one possible implementation, the processing module is specifically used to: obtain the refresh time and partition granularity specified by the user; determine the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information according to the refresh time and the partition granularity; and process the data within the refresh time corresponding to the data table of the job processing identified by each job identifier in the second task information according to the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information.
[0025] In one possible implementation, the processing module is further configured to: obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; the processing of the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information specifically includes: processing the data in the data table corresponding to the refresh time of the jobs identified by each job identifier in the second task information according to the scheduling frequency of the jobs identified by each job identifier in the second task information.
[0026] Thirdly, this application also provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the data processing method provided by the first aspect or any possible implementation thereof.
[0027] Fourthly, this application also provides a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the data processing method provided by the first aspect or any possible implementation thereof.
[0028] Fifthly, this application also provides a computer program product, including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the data processing method provided by the first aspect or any possible implementation thereof.
[0029] The aforementioned apparatus, computing device cluster, computer storage medium, or computer program product are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding solutions in the corresponding methods provided above, and will not be repeated here. Attached Figure Description
[0030] Figure 1 This is a schematic diagram of task information provided in an embodiment of this application;
[0031] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application;
[0032] Figure 3a and Figure 3b This is a schematic diagram illustrating a configuration of supplementary data information provided in an embodiment of this application;
[0033] Figure 4 and Figure 5 This is a schematic diagram illustrating an embodiment of updating task information provided in this application;
[0034] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;
[0035] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0036] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0037] Figure 9 This application provides an embodiment of a computing device cluster deployment Figure 6 A schematic diagram of the device shown. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.
[0039] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.
[0040] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals.
[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.
[0042] Before introducing the embodiments of this application, the terms mentioned in the embodiments of this application will be explained below.
[0043] Table lineage information describes the dependencies between data tables. A dependency is formed when data in one data table (called the destination table) originates from or is calculated from data in one or more other data tables (called source tables). The source table can be called the parent table of the destination table, and the destination table can be called the child table of the source tables. It's important to note that the terms "parent table" and "child table" are relative; a data table can be both the parent and child table of another data table.
[0044] For example, in big data scenarios, the initial data table undergoes different levels of processing to generate data tables at different levels. Each level of processing adds new features to the previous level's data table, thus generating a new data table. In other words, there is a lineage between data tables at different levels. For instance, the first-level processing might include adding first-level features to the initial data table to generate a first-level data table, the second-level processing might include adding second-level features to generate a new data table, and so on. The initial data table can include a data table obtained from the source database. The first-level data table refers to the new data table obtained after the first-level processing; the meanings of other levels are similar and will not be elaborated here. In some embodiments, the data tables at various levels in a big data scenario may include, but are not limited to, operational data store (ODS), data warehouse detail (DWD), data warehouse summary (DWS), and analytical data store (ADS) levels. In some embodiments, an ODS layer data table can be generated based on the source data table, then a DWD layer data table can be calculated based on the ODS layer data table, then a DWS layer data table can be calculated based on the DWD layer data table, and finally an ADS data table can be calculated based on the DWD layer or DWS layer data table. Here, the ODS layer data table refers to a new data table generated by overlaying features from the ODS layer. The meanings of the DWD layer data table, DWS layer data table, and ADS data table are similar and will not be elaborated further here.
[0045] Job lineage information describes the execution order relationship between jobs, i.e., upstream and downstream relationships. A job is the carrier of the computational logic of a data table. Specifically, a job can be a program used to process data in a data table, and the specific processing procedures can include, but are not limited to, operations such as updating, deleting, and inserting. In practical applications, when the processing of multiple data tables cannot be achieved by a single job, the processing of these multiple data tables can be divided into different jobs according to certain partitioning rules, and these processing procedures can be executed by different jobs. After the processing of multiple data tables is divided, upstream and downstream relationships will exist between the jobs. The job lineage information of a job can specifically include upstream information and / or downstream information. Upstream information includes at least the job identifier that identifies the upstream job of this job, and downstream information includes at least the job identifier that identifies the downstream job of this job. For example, in big data scenarios, partitioning rules can include, but are not limited to, one or more of the following factors: data table generation hierarchy, data table lineage information, business type, and priority. When there is an upstream and downstream relationship between two jobs, the upstream job can also be called the parent job, and the downstream job can also be called the child job. It is worth noting that the terms upstream job and downstream job (and parent job and child job) are relative. A job can act as both an upstream job (parent job) and a downstream job (child job) of another job. The job identifier that identifies an upstream job can also be called a parent job identifier, and the job identifier that identifies an upstream job can also be called a child job identifier.
[0046] Data replenishment (or data refresh / update) jobs refer to operations that update data in big data scenarios under certain circumstances. For example, when business rules change or data errors occur in a big data scenario, data replenishment jobs can be used to update data tables. Data tables in big data scenarios can be divided into multiple partitions based on time. The partition granularity of a data table can be daily, monthly, or yearly. Different data tables can have different partition granularities. For data tables partitioned by time, daily routine big data jobs can process data from the most recent partition. For example, a big data job for a data table T with a specific daily partition might be executed at 3 AM daily, processing data from the most recent three days each time. When a problem is found in the data for a specific time period X in data table T, a data replenishment job can be used to update the data for that time period X. In other words, after executing the data replenishment job, the data for time period X in data table T will be updated.
[0047] The partition field in a data table is used to indicate the granularity of partitioning. In big data scenarios, the partition granularity of a data table can be determined based on the value of the partition field, i.e., whether it is a daily, monthly, or yearly partition. For example, if the partition field value is in the format yyyy-mm-dd (yyyy represents year, mm represents month, and dd represents day), the partition granularity of the data table can be determined as a daily partition. Similarly, if the partition field value is in the format yyyy-mm, the partition granularity can be determined as a monthly partition. Finally, if the partition field value is in the format yyyy, the partition granularity can be determined as a yearly partition.
[0048] A directed acyclic graph (DAG) is a visual representation of task information. Task information describes the execution order of various jobs. Each node in a job DAG represents a job, and directed edges between nodes represent the execution order relationship between jobs. That is, if a directed edge exists between two nodes in a job DAG, it indicates that there is an execution order relationship between the jobs represented by those two nodes. If there is an execution order relationship between two jobs, the job executed earlier can be called the upstream job of the job executed later, and the job executed later can be called the downstream job of the job executed earlier. The directionality of a job DAG ensures that the execution order of jobs is explicit. The acyclicity of a job DAG ensures that there are no loops in the execution process, avoiding problems such as deadlock.
[0049] In big data scenarios, due to the existence of table lineages, updating one table also requires updating its child tables. This updating process necessitates the creation of jobs that process these tables. Executing some jobs also triggers downstream jobs. In other words, when data needs to be supplemented in a big data scenario, a large number of jobs are executed. To effectively manage and execute these jobs, relevant technologies utilize job directive AG orchestration and define the execution order relationships between jobs.
[0050] Figure 1 This is a visual illustration of task information in a big data scenario provided in an embodiment of this application. For example... Figure 1 As shown, task information is represented by a job DAG. The nodes in this job DAG represent jobs A1-A2, B1-B2, C1-C3, and D1-D3.
[0051] Figure 1The arrowed line segments connecting the tasks shown indicate the execution order between the two tasks. For example, task A1 and task A2 are connected by an arrowed line segment, indicating that task A1 is executed first, followed by task A2. Task A1 can be called the upstream task of task A2, and task A2 can be called the downstream task of task A1. Figure 1 The meanings of the other tasks shown and the arrowed line segments they connect are similar and will not be repeated here.
[0052] However, the arrangement Figure 1 The DAG of jobs shown has many considerations, which may add unnecessary upstream and downstream relationships between jobs in the DAG. This will not only consume valuable resources (such as computing and storage resources) but also increase the time of the entire data processing flow, thus leading to a decrease in data processing efficiency.
[0053] Therefore, this application provides a data processing method that can solve the above problems.
[0054] In the method provided in this application embodiment, after providing the user with the first task information of the data processing platform, the method obtains the start and end jobs specified by the user based on the first task information, determines the correlation between the start and end jobs specified by the user, removes irrelevant jobs between the start and end jobs, obtains the second task information, and performs data processing based on the second task information. This method can reduce the number of jobs that need to be executed, reduce resource consumption, and avoid increasing data processing time, thereby improving data processing efficiency.
[0055] The following is combined with Figure 2 The data processing method provided in the embodiments of this application will be described in detail.
[0056] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application. Figure 2 The method shown can be applied to data processing platforms in big data scenarios, and can be executed by one computing device in the data processing platform or by multiple computing devices working together.
[0057] like Figure 2 As shown, the method may include S201 to S204. The following example demonstrates execution using a computing device. Figure 1 right Figure 2 The steps involved in the method are described below. This is performed collaboratively by multiple computing devices. Figure 2 The methods shown include steps in which each computing device can perform [the task]. Figure 2 The method shown includes one or more steps.
[0058] S201 provides users with the first task information of the data processing platform.
[0059] The computing device of the data processing platform can obtain pre-stored first task information from the storage device in the data processing platform or from the storage space of the computing device, and then provide the first task information in a visual manner through the display interface, or provide the first task information to the user through the application programming interface.
[0060] The first task information is structured data. Specifically, the first task information may include multiple job identifiers, as well as upstream and / or downstream information corresponding to each job identifier. The multiple job identifiers are used to identify each job. The upstream and / or downstream information corresponding to each job identifier reflects the job execution order relationship between the jobs. Each job is used to process one or more data tables on the data processing platform.
[0061] by Figure 1 Taking the DAG of the tasks shown as an example, the first task information includes multiple task identifiers, which can be the identifiers of tasks A1~A2, B1~B2, C1~C3, and D1~D3.
[0062] In this context, tasks A1 and A2 are the starting tasks. There are no upstream tasks, but there are downstream tasks. That is, the identifiers of tasks A1 and A2 have corresponding downstream information, but no corresponding upstream information (or the upstream information is empty). Taking task A1 as an example, the downstream information of task A1 can include the identifier of task B1.
[0063] Among them, tasks B1-B2 and C1-C3 are intermediate tasks, with upstream and downstream components. That is, the identifiers of tasks B1-B2 and C1-C3 have corresponding upstream and downstream information. Taking task B1 as an example, its upstream information can include the identifier of task A1, and its downstream information can include the identifiers of tasks C1 and C2. Taking task C2 as an example, its upstream information can include the identifiers of tasks B1 and B2, and its downstream information can include the identifier of task D2.
[0064] Among them, tasks D1 to D3 are completed tasks, with upstream tasks but no downstream tasks. That is, the identifiers of tasks D1 to D3 have corresponding upstream information, but no corresponding downstream information (or the downstream information is empty). Taking task D3 as an example, the upstream information of task D3 can include the identifier of task C3.
[0065] The first task information can be generated based on the job lineage information of the data processing platform. In this case, the upstream job of a job identified by a job identifier is also called the parent job of that job, and the downstream job of that job is also called the child job of that job.
[0066] S202, obtain the user-specified start job and end job, where the start job and end job are one or more jobs identified by multiple job identifiers in the first task information.
[0067] In big data scenarios, users can provide supplementary data information to the computing devices of a data processing platform through their terminal devices. The user's terminal device can have a client application for the data processing platform deployed on it. When a user needs supplementary data, they can generate the supplementary data information through this client and then send it to the data processing platform through their terminal device. Specifically, the user can run the client on their terminal device, access the supplementary data page through the client, and configure the supplementary data information there.
[0068] Specifically, supplementary data information may include start information and end information. The start information may include the start data table identifier and / or the start job identifier. The end information may include the end data table identifier, the end job identifier, and one or more of the following: select all downstream task information. It is understood that the supplementary data information may include one or more identifiers such as the start data table identifier, start job identifier, end data table identifier, and end job identifier. Figure 3a As shown, after entering the supplementary data page, users can select the starting data table, the starting job, the ending data table, the ending job, and all downstream jobs of the starting job.
[0069] In some embodiments, the supplementary data information may further include values of functional parameters. These functional parameters indicate whether irrelevant jobs are removed. For example... Figure 3b As shown, after entering the supplementary data page, the user can select the starting data table, the starting job, the ending data table, the ending job, and all downstream jobs of the starting job. The user can also configure the values of functional parameters. When the user-configured functional parameter value indicates that irrelevant jobs should not be removed, the computing device may not execute S202 and S203.
[0070] The computing device can determine the start and end jobs specified by the user based on the received supplementary data information.
[0071] Taking a scenario where the starting information includes a starting job identifier and the ending information includes an ending job identifier as an example, the computing device can query the first task information of the data processing platform based on the starting job identifier and the ending job identifier to obtain the job between the starting job and the ending job. For example: querying the first task information based on the starting job identifier yields the upstream and downstream information corresponding to the starting job identifier. Then, the device continues to query the first task information based on the job identifiers contained in the upstream and downstream information corresponding to the starting job identifier. The query ends when the job identifiers contained in the continuously queried upstream and downstream information are the same as the ending job identifier.
[0072] Taking an example where the starting information includes a starting data table identifier and the ending information includes an ending data table identifier, the computing device can use the job that processes the data table identified by the starting data table identifier as the starting job and the job that processes the data table identified by the ending data table identifier as the ending job. Then, based on the starting job identifier and the ending job identifier, it queries the first task information to obtain the jobs between the starting job and the ending job. The specific process by which the computing device determines the jobs between the starting job and the ending job can be referred to the explanation in the previous paragraph, and will not be repeated here. The specific process by which the computing device determines the starting job and the ending job may include: the computing device can query the table lineage information of the starting data table based on the starting data table identifier to obtain the starting job identifier; similarly, it can query the table lineage information of the ending data table based on the ending data table identifier to obtain the ending job identifier.
[0073] Taking the starting information as including the starting job identifier and the ending information as including the selection of all downstream task information as an example, the computing device can query the first task information based on the starting job identifier to obtain the upstream and downstream information corresponding to the starting job identifier. Then, it can continue to query the first task information based on the upstream and downstream information corresponding to the starting job identifier until the upstream and downstream information of the data processing platform has been queried.
[0074] S203, determine the correlation between the starting task and the ending task, remove the irrelevant tasks between the starting task and the ending task to obtain the second task information.
[0075] Specifically, the computing device can traverse all jobs between the starting job and the ending job, and based on the table lineage information of the data tables processed by each job between the starting job and the ending job, as well as the upstream information of each job between the starting job and the ending job, determine the unrelated jobs between the starting job and the ending job, remove the unrelated jobs between the starting job and the ending job, and obtain the second task information. The upstream information corresponding to each of the multiple job identifiers includes the upstream information of each job between the starting job and the ending job.
[0076] The process by which a computing device determines unrelated jobs between a starting job and an ending job may include: taking the current job as an example, among the jobs between the starting and ending jobs, the upstream jobs of the current job are determined based on the upstream information of the current job. If the data table processed by the current job has no table lineage relationship with the data tables processed by the upstream jobs of the current job, the current job is considered an unrelated job. Specifically, the table lineage information of the data table processed by the current job includes a job list, which includes one or more job identifiers, and the data table processed by the job identifier in the job list is the parent data table of the data table processed by the current job. If any job identifier in the upstream information of the current job is different from the job identifier in the job list, it indicates that the upstream information of the current job and the job list have no intersection, that is, the data table processed by the current job has no table lineage relationship with the data tables processed by the upstream jobs of the current job, and the current job is then considered an unrelated job. In some possible implementations, the current job and all its downstream jobs may also be considered unrelated jobs.
[0077] For example, taking the initial task as Figure 1 Taking job A1 as an example, the downstream jobs of job A1 include job B1, job C2, job D2, job C1, and job D1. Taking job B1 as the current job, if the job lists included in the table lineage information of the data tables processed by job A1 and job B1 do not overlap, job B1 can be considered an unrelated job. Taking job C2 as the current job, if the job lists included in the table lineage information of the data tables processed by job B1 and job B2 do not overlap, job C2 can be considered an unrelated job.
[0078] The computing device removes irrelevant jobs between the starting and ending jobs to obtain second task information. Specifically, this may include removing the job identifiers and their corresponding upstream / downstream information of irrelevant jobs from the first task information. Taking all downstream jobs of the starting job, including the current job, as an example, the job identifier of the current job and its corresponding upstream / downstream information can be removed from the first task information.
[0079] Furthermore, if the data table processed by the current job has no table lineage relationship with the data table processed by an upstream job, the job identifier identifying that upstream job can be removed from the upstream information of the current job. This removes the upstream / downstream relationship between the current job and that upstream job, yielding the second task information. For example, for job C2, if the job identifiers in the job lists included in the table lineage information of the data tables processed by job B1 and job C2 are all different, such as... Figure 4As shown, the job identifier of job B1 in the upstream information of job C2 can be removed, thus eliminating the upstream / downstream relationship between job B1 and job C2. In other words, the updated upstream information for job C2 does not include the job identifier of job B1.
[0080] Furthermore, when the upstream information corresponding to the updated current job is empty, both the current job and its downstream jobs can be removed. Figure 1 As shown in the example, job C2 is the current job. Figure 5 As shown, if the upstream and downstream relationships between jobs B1 and B2 and job C2 are removed, then job C2 and its downstream job D2 can be removed. Removing job C2 and its downstream job D2 can include removing job C2 and its downstream information, as well as job D2 and its upstream information, from the first task information.
[0081] S204, perform data processing based on the second task information.
[0082] After obtaining the second task information, the computing device can execute each task sequentially according to the task identifier and its corresponding upstream and downstream information included in the second task information.
[0083] Users can also configure the refresh time on the data replenishment page. In this case, the computing device can process data according to the user-specified refresh time.
[0084] Specifically, the computing device can obtain the refresh time specified by the user, and determine the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information based on the refresh time and the partition granularity of the data tables of the jobs identified by each job identifier in the second task information; and process the data within the refresh time corresponding to the data tables of the jobs identified by each job identifier in the second task information according to the number of instances corresponding to the data tables of the jobs identified by each job identifier in the second task information.
[0085] Specifically, the computing device can also obtain the refresh time and partition granularity specified by the user; based on the refresh time and the partition granularity, determine the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information; and process the data within the refresh time corresponding to the data table of the job processing identified by each job identifier in the second task information according to the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information.
[0086] Specifically, the computing device can also obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; the processing of the data in the data table of the jobs identified by each job identifier in the second task information within the refresh time specifically includes: processing the data in the data table of the jobs identified by each job identifier in the second task information within the refresh time according to the scheduling frequency of the jobs identified by each job identifier in the second task information.
[0087] In big data scenarios, jobs are typically executed daily, meaning the execution cycle is one day. However, the partitioning granularity of the data tables processed by these jobs can be daily, monthly, or yearly. In related technologies, when data needs to be supplemented, daily jobs generate a new instance each day based on the time range to be updated. For monthly and yearly partitioned tables, this results in a large amount of redundant computation and resource consumption. Therefore, computing devices can determine the number of instances N based on the partitioning granularity of the data table to be processed by the job, and then process the data in the table using N instances. Determining the number of instances according to the partitioning granularity of the data table reduces redundant instances and further improves data processing efficiency.
[0088] Specifically, the computing device can determine the number of instances corresponding to the data table of each job identified by the job identifier in the second task information, which may include: when the partition granularity of the data table is daily, the number of instances N is determined by the refresh time per day; when the partition granularity of the data table is monthly, the number of instances N is determined by the refresh time per month; when the partition granularity of the data table is yearly, the number of instances N is determined by the refresh time per year. Taking a refresh time of 01 / 01-02 / 29 of a certain year as an example, when the partition granularity of the data table is daily, the number of instances N is 60; when the partition granularity of the data table is monthly, the number of instances N is 2; and when the partition granularity of the data table is yearly, the number of instances N is 1.
[0089] Specifically, the partition granularity of the data table for each job identified by each job identifier in the second task information can be obtained from the attribute information of the job identified by each job identifier in the second task information. Furthermore, the partition granularity of the data table for each job identified by each job identifier in the second task information can also be determined based on the format of the partition values of the data tables for each job processing obtained from the data tables of each job processing.
[0090] In the above Figure 2 In the method embodiment shown, by determining the correlation between the user-specified start job and end job, and removing the irrelevant jobs between the start job and the end job, the number of jobs that need to be executed can be reduced, thereby improving data processing efficiency.
[0091] based on Figure 2The method embodiments shown in this application also provide a data processing apparatus. This data processing apparatus can be used to perform the above-described... Figure 2 The steps in the method shown.
[0092] Figure 6 This is a schematic diagram of the structure of a data processing device 600 provided in an embodiment of this application. Figure 6 As shown, the data processing device 600 may include a providing module 601, an acquiring module 602, and a processing module 603. When the data processing device 600 is used to perform the above... Figure 2 The functions of each module in each step of the method shown are described below.
[0093] The providing module 601 is used to provide the user with first task information of the data processing platform. The first task information includes multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information are used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform.
[0094] The acquisition module 602 is used to acquire the start job and end job specified by the user, wherein the start job and end job are one or more jobs among the various jobs, determine the correlation between the start job and end job, and remove irrelevant jobs between the start job and end job to obtain the second task information.
[0095] The processing module 603 is used to perform data processing based on the second task information.
[0096] It should be noted that, Figure 6 The data processing apparatus 600 provided in the illustrated embodiment, when executing the data processing method, is only illustrated by the division of the above-described functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the computing device provided in the above embodiment and... Figure 2 The data processing method embodiments shown belong to the same concept, and their specific implementation process can be found in the method embodiments and invention content, which will not be repeated here.
[0097] When the above-described module is used as an example of a software functional unit, the data processing device 600 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Further, the aforementioned computing device may be one or more. For example, the data processing device 600 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0098] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0099] When the above-mentioned module is used as an example of a hardware functional unit, the module may include at least one computing device, such as a server. Alternatively, the module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0100] The multiple computing devices included in a module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in a module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in a module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0101] It should be noted that, in other embodiments, the providing module 601, the obtaining module 602, and the processing module 603 can be used to perform... Figure 2 Any step in the method shown. The steps implemented by the providing module 601, the obtaining module 602, and the processing module 603 can be specified as needed, and implemented by the providing module 601, the obtaining module 602, and the processing module 603 respectively. Figure 2 The different steps in the method shown enable the full functionality of the data processing device 600.
[0102] This application also provides a computing device 700. For example... Figure 7 As shown, the computing device 700 includes a processor 701, a memory 702, a communication interface 703, and a bus 704. The processor 701, memory 702, and communication interface 703 communicate with each other via the bus 704. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.
[0103] The 704 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 104 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 702, processor 701, communication interface 703).
[0104] Processor 701 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0105] The memory 702 may include volatile memory, such as random access memory (RAM). The processor 701 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0106] The memory 702 stores executable program code, and the processor 701 executes the executable program code to implement the functions of the aforementioned providing module 601, obtaining module 602, and processing module 603, thereby achieving... Figure 2 The method shown. That is, the memory 702 stores the information for executing... Figure 2 The instructions for the method shown. And / or, executable code is stored in memory 702, and processor 701 executes this executable code to implement the functions of the aforementioned providing module 601, obtaining module 602, and processing module 603, thereby achieving... Figure 7 The method shown. That is, the memory 702 stores the information for executing... Figure 7 The instructions for the method shown.
[0107] The communication interface 703 uses modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 700 and other devices or communication networks.
[0108] based on Figure 2 The method embodiments shown in this application also provide a computing device cluster.
[0109] Figure 8 This application provides a computing device cluster 800.
[0110] The computing device cluster 800 may include the data processing platform described above. The computing device cluster includes at least one computing device. This computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0111] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 702 of one or more computing devices 700 in the computing device cluster may store the same memory for executing... Figure 2 The instructions for the method shown.
[0112] In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster may also store memory for execution. Figure 2 The instructions of the method shown are partial. In other words, a combination of one or more computing devices 700 can jointly execute instructions for performing... Figure 2 The instructions for the method shown.
[0113] It should be noted that the memories 702 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device 600. That is, the instructions stored in the memories 702 of different computing devices 700 can implement the functions of one or more modules among the providing module 601, the acquiring module 602, and the processing module 603.
[0114] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, the two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 702 in computing device 700A stores instructions for performing the functions of the providing module 601 and the obtaining module 602. Meanwhile, the memory 702 in computing device 700B stores instructions for performing the functions of the processing module 603. Figure 9 The connection method between the computing device clusters shown can be considered because the data processing method provided in this application requires a large amount of storage and computing resources (e.g., storing a large amount of data), therefore, the functions implemented by the processing module 603 are considered to be executed by the computing device 700B. It should be understood that... Figure 9The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.
[0115] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster. The difference is that the memory 702 of one or more computing devices 700 in this computing device cluster can store the same information for execution. Figure 2 The instructions for the data processing method shown.
[0116] In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster may also store memory for execution. Figure 2 The instructions for the data processing method shown are partial instructions. In other words, a combination of one or more computing devices 700 can jointly execute instructions for performing data processing. Figure 2 The instructions for the data processing method shown.
[0117] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform... Figure 2 The data processing method shown.
[0118] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute... Figure 2 The data processing method shown, or instructions to the computing device to perform Figure 2 The data processing method shown.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data processing method, characterized by, Applied to a data processing platform, the method includes: The data processing platform provides users with first task information, which includes multiple job identifiers and upstream and / or downstream information corresponding to each job identifier. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information is used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform. Obtain the start job and end job specified by the user, wherein the start job and end job are one or more of the various jobs; Determine the correlation between the starting task and the ending task, and remove irrelevant tasks between the starting task and the ending task to obtain the second task information; Data processing is performed based on the second task information.
2. The method of claim 1, wherein, The method further includes: Retrieve the user-specified start and end data tables; The starting job is determined based on the starting data table, and the ending job is determined based on the ending data table.
3. The method according to claim 1 or 2, characterized in that, The step of determining the correlation between the starting job and the ending job, and removing irrelevant jobs between the starting job and the ending job to obtain the second task information specifically includes: Based on the table lineage information of the data tables of each job processing between the starting job and the ending job, and the upstream information of each job between the starting job and the ending job, unrelated jobs between the starting job and the ending job are determined. The upstream information corresponding to each job identifier in the plurality of job identifiers includes the upstream information of each job between the starting job and the ending job. Remove irrelevant tasks between the start and end tasks to obtain the second task information.
4. The method according to any one of claims 1-3, characterized in that, The data processing based on the second task information includes: Get the refresh time specified by the user; The data corresponding to the refresh time in the data table of each job identifier in the second task information is processed.
5. The method according to any one of claims 1-3, characterized in that, The data processing based on the second task information includes: Get the refresh time specified by the user; Based on the refresh time and the partition granularity of the data table processed by each job identifier in the second task information, determine the number of instances corresponding to the data table processed by each job identifier in the second task information. Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.
6. The method according to any one of claims 1-3, characterized in that, The data processing based on the second task information includes: Obtain the refresh time and partition granularity specified by the user; Based on the refresh time and the partition granularity, determine the number of instances corresponding to the data table processed by each job identifier in the second task information; Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.
7. The method according to any one of claims 4-6, characterized in that, The method further includes: Obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; The processing of the data within the refresh time corresponding to each job identifier in the data table of the second task information specifically includes: According to the scheduling frequency of the jobs identified by each job identifier in the second task information, the data within the refresh time corresponding to the data table of the jobs processed by each job identifier in the second task information is processed.
8. A data processing apparatus, characterized in that, The device, applied to a data processing platform, includes: A providing module is used to provide users with first task information of the data processing platform. The first task information includes multiple job identifiers and upstream and / or downstream information corresponding to each of the multiple job identifiers. The multiple job identifiers are used to identify each job, and the upstream and / or downstream information are used to reflect the relationship of the job execution order between the jobs. Each job is used to process one or more data tables on the data processing platform. The acquisition module is further configured to acquire the user-specified start job and end job, wherein the start job and end job are one or more jobs among the various jobs, determine the correlation between the start job and end job, and remove irrelevant jobs between the start job and end job to obtain second task information; The processing module is used to process data based on the second task information.
9. The apparatus according to claim 8, characterized in that, The acquisition module is also used for: Retrieve the user-specified start and end data tables; The starting job is determined based on the starting data table, and the ending job is determined based on the ending data table.
10. The apparatus according to claim 8 or 9, characterized in that, The acquisition module is also used for: Based on the table lineage information of the data tables of each job processing between the starting job and the ending job, and the upstream information of each job between the starting job and the ending job, unrelated jobs between the starting job and the ending job are determined. The upstream information corresponding to each job identifier in the plurality of job identifiers includes the upstream information of each job between the starting job and the ending job. Remove irrelevant tasks between the start and end tasks to obtain the second task information.
11. The apparatus according to any one of claims 8-10, characterized in that, The processing module is specifically used for: Get the refresh time specified by the user; The data corresponding to the refresh time in the data table of each job identifier in the second task information is processed.
12. The apparatus according to any one of claims 8-10, characterized in that, The processing module is specifically used for: Get the refresh time specified by the user; Based on the refresh time and the partition granularity of the data table processed by each job identifier in the second task information, determine the number of instances corresponding to the data table processed by each job identifier in the second task information. Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.
13. The apparatus according to any one of claims 8-10, characterized in that, The processing module is specifically used for: Obtain the refresh time and partition granularity specified by the user; Based on the refresh time and the partition granularity, determine the number of instances corresponding to the data table processed by each job identifier in the second task information; Based on the number of instances corresponding to the data table of the job processing identified by each job identifier in the second task information, the data within the refresh time specified in the data table of the job processing identified by each job identifier in the second task information is processed.
14. The apparatus according to any one of claims 11-13, characterized in that, The processing module is also used for: Obtain the scheduling frequency of the jobs identified by each job identifier in the second task information specified by the user; The processing of the data within the refresh time corresponding to each job identifier in the data table of the second task information specifically includes: According to the scheduling frequency of the jobs identified by each job identifier in the second task information, the data within the refresh time corresponding to the data table of the jobs processed by each job identifier in the second task information is processed.
15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method according to any one of claims 1-7.
16. A computer-readable storage medium, characterized in that, Includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-7.
17. A computer program product, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-7.