Data processing method, electronic device, computer storage medium and program product

Abstract

The embodiment of the present application provides a data processing method, an electronic device, a computer storage medium and a program product, wherein the data processing method includes: obtaining a directed graph corresponding to the offline computing logic for the target data; grouping the directed graph according to the task instance label of each node in the directed graph, wherein the task instance label is used to identify the historical task instance where the node's reusable task instance result is located; calculating at least one weakly connected component in the directed graph portion corresponding to each group, selecting the target historical task instance indicated by the task instance label corresponding to the group with the weakly connected component in the front row, so as to reuse the task instance result of the target historical task instance. The data processing scheme provided by the embodiment of the present application is more universal, and can greatly increase the number of nodes for reusing task instance results, reducing the required computing resources and storage resources for storing the computing results.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data processing method, an electronic device, a computer storage medium, and a program product. Background Art

[0002] Offline computing is mainly used in big data modeling platforms. Big data modeling platforms provide users with editing capabilities. Users can edit offline computing logic corresponding to offline computing models according to their own needs. The big data modeling platform can schedule and execute directed graphs corresponding to offline computing models edited by users to complete users' offline computing tasks.

[0003] Generally, an offline calculation model established by a user will be repeatedly executed. In order to reduce the amount of calculation, offline calculation is mainly performed on incremental data. For example, the offline calculation model is used to perform offline calculations on the data of the past 30 days, so as to obtain the statistical data of the past 30 days; if the user needs to obtain the statistical data of the past 30 days every day, in order to reduce the amount of calculation, the incremental calculation can be mainly performed on the newly added data every day, and then combined with the previous calculation results to obtain the statistical data.

[0004] However, the above process is only applicable to offline computing scenarios where incremental data can be clearly identified, and its applicable scenarios are limited. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a data processing solution to at least partially solve the above-mentioned problem.

[0006] According to the first aspect of an embodiment of the present application, a data processing method is provided, including: obtaining a directed graph corresponding to an offline computing logic for target data; grouping the directed graph according to task instance labels of each node in the directed graph, wherein the task instance labels are used to identify historical task instances where reusable task instance results of nodes are located; calculating at least one weakly connected component in the directed graph portion corresponding to each group, and selecting a target historical task instance indicated by the task instance label corresponding to the group in which the weakly connected component is located in the front column, so as to reuse the task instance results of the target historical task instance.

[0007] According to a second aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data processing method described above is implemented.

[0008] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the data processing method described above.

[0009] According to the data processing scheme provided in the embodiment of the present application, a directed graph corresponding to the offline computing logic for the target data is obtained, and the directed graph is grouped according to the task instance tags of each node in the directed graph, wherein the task instance tags are used to identify the historical task instances where the reusable task instance results of the nodes are located, at least one weakly connected component in the directed graph portion corresponding to each group is calculated, and the target historical task instance indicated by the task instance tag corresponding to the group with the weakly connected component in the front row is selected to reuse the task instance results of the target historical task instance. Thus, the task instance results in the reused historical task instances can be determined based on the nodes instead of the incremental data, so that the data processing method provided in the present embodiment is not limited by the incremental data, is more universal, and has a wider range of applications. In addition, in the present embodiment, the task instance results in the reused historical task instances are determined based on the nodes, which can greatly increase the number of nodes for reusing the task instance results, and further reduce the required computing resources and storage resources for storing the computing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0011] Figure 1A A flowchart of a data processing method according to Embodiment 1 of the present application;

[0012] Figure 1B for Figure 1A A schematic diagram of an example scenario in the illustrated embodiment;

[0013] Figure 2A A flowchart of a video data processing method provided in an embodiment of the present application;

[0014] Figure 2B for Figure 2A A directed graph with task instance labels in the illustrated embodiment;

[0015] Figure 2C for Figure 2A A flowchart of the steps of a method for determining a task instance label in the illustrated embodiment;

[0016] Figure 2D for Figure 2A A directed graph of reused historical task instances in the illustrated embodiment;

[0017] Figure 3A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0020] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0021] Figure 1A A schematic flow chart of a data processing method provided by this embodiment is shown, as shown in the figure, which includes:

[0022] S101. Obtain a directed graph corresponding to offline computing logic for target data.

[0023] In this embodiment, the user of the big data platform can edit the offline computing model corresponding to the offline computing logic according to their own needs. The offline computing model can be a directed graph corresponding to the offline computing logic. It should be noted that an offline computing model can include one or more directed graphs, which is not limited in this embodiment.

[0024] A graph can be defined as a set G = {V, E} of nodes V and edges E, where V represents the set of all nodes in the graph, and E represents the set of all edges in the graph. A directed graph is a graph in which the edge elements in the set E have directions.

[0025] The directed graph corresponding to the offline computing logic includes multiple nodes, each of which may correspond to a computing rule provided by a big data platform. The node may specifically be an SQL script, a SHELL script, a PYTHON script, etc., which is not limited in this embodiment.

[0026] Multiple nodes in a directed graph are connected by directed edges. Among the nodes connected by directed edges, the output of the upstream node is the input of the downstream node. For example, nodes A and B are connected by directed edge C, and directed edge C points from node A to node B, then the output of node A is the input of node B.

[0027] S102. Grouping the directed graph according to the task instance label of each node in the directed graph, wherein the task instance label is used to identify the historical task instance where the node's reusable task instance result is located.

[0028] In this embodiment, each time the target data is calculated offline through the offline calculation model, a task instance corresponding to the offline calculation model will be established first. The task instance can record the operation record of this calculation process, for example, the reference data address corresponding to the calculation process, the result data address, the interface called by the calculation process, etc.

[0029] The task instance tag corresponding to the node is used to identify the historical task instance where the task instance result that the node can reuse is located. Specifically, the historical task instances can be numbered in sequence according to the execution order, marked as 1, 2, 3, etc., and the historical task instances can be identified by the number.

[0030] In this embodiment, the directed graph is grouped according to the task instance label, specifically, the nodes in the directed graph are grouped according to the task instance label; after grouping, the nodes in each group of corresponding directed graph parts can reuse the task instance results in the same historical task instance.

[0031] S103. Calculate at least one weakly connected component in the directed graph portion corresponding to each group, select the target historical task instance indicated by the task instance mark corresponding to the group with the weakly connected component in the front column, so as to reuse the task instance result of the target historical task instance.

[0032] In this embodiment, after the nodes are grouped, the grouped nodes can be reorganized into a directed graph based on the upstream and downstream relationships of the nodes in the directed graph to obtain a partial directed graph corresponding to the grouping, and the weakly connected components of the directed graph portion corresponding to the grouping can be calculated.

[0033] Specifically, a weakly connected component of a directed graph is a maximal subgraph in which any two points in the directed graph are connected by a directed edge. A weakly connected component being in the front column means that after the weakly connected components are sorted according to the number of nodes, a certain weakly connected component is in the front column in the sorting.

[0034] Since the nodes in each group can reuse the task instance results in the same historical task instance, the target historical task instance indicated by the task instance mark corresponding to the group with the weakly connected component in the front column is selected. That is, from the historical task instances corresponding to multiple task instance marks, the one with more reusable task instance results is selected as the target historical task instance.

[0035] After the target historical task instance is determined, the calculation results of the nodes included in the group corresponding to the target task instance can be directly reused with the results in the target historical task instance, and offline calculation for the target data can be performed accordingly.

[0036] Therefore, the task instance results in the historical task instances reused in this offline computing process can be determined based on the nodes rather than based on the incremental data, so that the data processing method provided by this embodiment is not limited by incremental data, is more universal, and has a wider range of applications; in addition, in this embodiment, the task instance results in the reused historical task instances are determined based on the nodes, which can greatly increase the number of nodes for reusing the task instance results, further reducing the required computing resources and storage resources for storing the calculation results.

[0037] The solution of this application is exemplified below through a specific usage scenario.

[0038] See also Figure 1B , shows a directed graph corresponding to the offline computing logic for target data.

[0039] A directed graph includes multiple nodes, each of which can correspond to a calculation rule, such as Figure 1B As shown, the graph may include nodes.

[0040] Some nodes in the directed graph (e.g. nodes 1-9) correspond to task instance tags. Then, the directed graph can be grouped according to the task instance tags to obtain Figure 1B According to the weakly connected components of the directed graph corresponding to each group, we can determine Figure 1B Among the multiple partial directed graphs shown in , if the weakly connected component of the partial directed graph corresponding to the task instance label “1” is larger, the historical task instance corresponding to the task instance label “1” will be taken as the target task instance.

[0041] When performing calculations, the results of multiple nodes 1-5, 7, and 8 in the group corresponding to the task instance mark "1" can be directly reused with the task instance results in the target historical task instance indicated by the task instance mark "1", and the remaining nodes 6, 9, and 10 can be recalculated to complete the offline calculation process for the target data.

[0042] The solution provided by the present embodiment obtains a directed graph corresponding to the offline computing logic for the target data, and groups the directed graph according to the task instance tags of each node in the directed graph, wherein the task instance tags are used to identify the historical task instances where the reusable task instance results of the nodes are located, calculates at least one weakly connected component in the directed graph portion corresponding to each group, and selects the target historical task instance indicated by the task instance tag corresponding to the group with the weakly connected component in the front row, so as to reuse the task instance results of the target historical task instance. Thus, the task instance results in the reused historical task instances can be determined based on the nodes instead of the incremental data, so that the data processing method provided by the present embodiment is not limited by the incremental data, is more universal, and has a wider range of applications. In addition, in the present embodiment, the task instance results in the reused historical task instances are determined based on the nodes, which can greatly increase the number of nodes for reusing the task instance results, and further reduce the required computing resources and storage resources for storing the computing results.

[0043] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.) and a PC, etc.

[0044] Figure 2A A schematic flow chart of a data processing method provided by this embodiment is shown, as shown in the figure, which includes:

[0045] S201. Obtain a directed graph corresponding to the offline computing logic for target data.

[0046] The specific implementation of this step can refer to the above embodiment, which will not be repeated here. In any embodiment of the present application, the directed graph can be a directed acyclic graph (DAG), that is, a directed graph without loops.

[0047] S202. For the directed graph corresponding to the offline computing logic of the target data, generate a current task instance corresponding to the directed graph according to the target data.

[0048] The task instance includes current subtask instances corresponding one-to-one to the nodes of the directed graph.

[0049] In any embodiment of the present application, when it is determined to perform offline calculations, a corresponding task instance can be generated. The task instance can include multiple subtask instances. The subtask instances can correspond one-to-one to the nodes in the directed graph. The subtask instances are used to record the running records of the node calculations.

[0050] S203. According to the current subtask instance corresponding to the node of the reusable task instance result, determine the reusable task instance result of the current subtask instance from the historical subtask instances corresponding to the node.

[0051] In any embodiment of the present application, the node of the reusable task instance result can be determined first, and this step and subsequent steps can be performed on the determined node.

[0052] Specifically, a node with idempotence may be determined as a node that can reuse the result of a task instance.

[0053] A node with idempotency refers to a node that has the same effect after any multiple executions. For example, a node with idempotency may be a SQL query node, and the effect of any multiple executions is consistent when the queried data and SQL do not change.

[0054] In addition, the operation node can also be determined as a node of a non-reusable task instance result. The operation node refers to a node that contains an operation that must be executed, such as data synchronization, interface call, etc.

[0055] Specifically, for the determined directed graph, you can traverse from the root node of the directed graph, determine the nodes of the reusable task instance results from the directed graph, and execute this step and subsequent steps for each node of the reusable task instance result.

[0056] When traversing, in order to facilitate the transmission of information such as task instance tags of upstream nodes to downstream nodes, depth-first traversal is preferably used. Of course, other node traversal schemes such as breadth-first traversal are also within the protection scope of this application.

[0057] In this embodiment, for the determined node of the reusable task instance result, the reusable task instance result of the current subtask instance can be determined from the historical subtask instances corresponding to the node.

[0058] Specifically, based on the current subtask instance corresponding to the node, a historical subtask instance with the same instance parameters as the current subtask instance and a consistent reference table can be determined from the historical subtask instances corresponding to the node, and its result can be determined as a reusable task instance result.

[0059] For example, if the node is an SQL node, the instance parameters of the historical subtask instance and the current subtask instance are the same, which may be that the SQL scripts or the simplest syntax trees corresponding to the two subtask instances are consistent.

[0060] In this embodiment, the reference table refers to the data table referenced by the node during the calculation process, and the consistency of the reference table means that the referenced table has not changed. Specifically, the consistency of the reference table can be determined by comparing the table structure of the reference table of the two task instances and the modification time of the reference table.

[0061] For example, if the target data for offline calculation is multiple tables, and the purpose of offline calculation is to aggregate the multiple tables, then the reference table may be the multiple tables included in the target data.

[0062] In addition to the above two conditions, it is also necessary to determine whether the life cycle of the result table of the historical subtask instance is greater than the threshold. If it is greater than the threshold, the result is determined to be a reusable task instance result, avoiding the situation where this step determines that it can be reused, but when the task instance result is reused later, the task instance result cannot be read due to the end of the life cycle of the result table.

[0063] S204. Add a task instance tag to the node according to the determined historical task instance where the reusable task instance result is located.

[0064] In this embodiment, after determining the task instance labels of some or all nodes, the method also includes: for nodes with upstream and downstream relationships in a directed graph, if the task instance label of the downstream node does not correspond to the upstream node of the downstream node, the task instance label corresponding to the downstream node is deleted.

[0065] If the upstream node cannot reuse the task instance result in the historical task instance, it means that the calculation process of the upstream node or the data associated with the calculation process has changed, that is, the calculation result of the upstream node has changed; and the calculation result of the upstream node is the input of the downstream node. Therefore, the downstream node cannot use the task instance result in the historical task instance. The task instance tag corresponding to the downstream node must be a subset of the task instance tag corresponding to the upstream node. In this case, the task instance tag corresponding to the historical task instance shall be deleted from the task instance tag list corresponding to the downstream node.

[0066] See also Figure 2B , shows a directed graph with task instance tags added, including 2 directed graphs with a total of 13 nodes, Figure 2B The DataSource node in the figure refers to the data source node, the filter node in the figure is a data filtering node, the JOHN node in the figure is a data merging node, the distinct node in the figure is a screening node, the GROUP_BY node in the figure is a data integration node for grouping and integrating data, and the DATA_SYNC in the figure is a data synchronization node.

[0067] Assuming that the offline model history has been executed 4 times, the offline model corresponds to 4 historical task instances, and the corresponding task instance labels are 1, 2, 3, and 4. The curly brackets above each node are the task instance labels of the node. As can be seen from the figure, some nodes in the directed graph may have task instance labels, while others may not.

[0068] Figure 2B A strikethrough is added to the task instance tag to be deleted. For example, the task instance tag corresponding to the second node "filter" includes "1,2,3,4". However, the task instance tag of the parent node "data_source" of the second node does not include "4". Therefore, the task instance tag "4" of the second node is deleted.

[0069] See also Figure 2C , steps S201 - S204 are exemplarily described through an application scenario.

[0070] A) After obtaining the directed graph, the traversal process starts from the root node of the directed graph.

[0071] B) For the current node, determine whether it is a SQL query node.

[0072] If yes, continue to execute step C; if no, execute step F to continue traversing the next node.

[0073] In this step, the node that can reuse the task instance result is pre-set as the SQL query node, and other nodes cannot reuse the task instance result. Of course, the node that can reuse the result here can be set by the user according to the needs, and this embodiment does not limit this.

[0074] C) Obtain multiple historical subtask instances corresponding to the node, and traverse the multiple historical subtask instances one by one. For the jth historical subtask instance, determine:

[0075] 1. Whether the instance parameters of the current subtask instance of the node are consistent with those of the historical subtask instance;

[0076] 2. Whether the reference table of the current subtask instance of the node is consistent with that of the historical subtask instance;

[0077] 3. Whether the life cycle of the result table of the historical subtask instance is greater than the threshold.

[0078] D) If all three conditions are met, the historical subtask instance is added to the node's reusable list.

[0079] E) After all historical subtask instances are traversed, the historical task instances where the historical subtask instances are located are determined and deduplicated, and corresponding task instance labels are added to the nodes based on the deduplicated historical task instances.

[0080] F) Determine whether all nodes of the directed graph have been traversed.

[0081] If completed, execute step G. If not completed, repeat steps BF for the next node until the node traversal is completed.

[0082] G) Perform label simplification.

[0083] If the task instance tag of the downstream node is not a subset of the task instance tag of the upstream node, the extra task instance tags in the downstream node are deleted.

[0084] After execution, you can get the above Figure 2B .

[0085] S205 . Divide the nodes in the directed graph into multiple groups according to the task instance labels of the nodes.

[0086] In this embodiment, nodes with multiple task instance labels are located in multiple groups.

[0087] For example, if a node corresponds to three task instance tags, the node is simultaneously assigned to the groups corresponding to the three task instance tags.

[0088] S206. Connect the nodes in each group according to the directed edges between the nodes in the directed graph to obtain a partial directed graph corresponding to the group.

[0089] For example, for each group, the nodes in each group may be connected according to the directed edges between the nodes in the directed graph, so as to reconstruct a partial directed graph corresponding to the group.

[0090] S207. Calculate at least one weakly connected component in the directed graph portion corresponding to each group, select the target historical task instance indicated by the task instance mark corresponding to the group with the weakly connected component in the front column, so as to reuse the task instance result of the target historical task instance.

[0091] In any embodiment of the present application, the specific method of calculating the weakly connected components can refer to the relevant technology and will not be repeated here.

[0092] After calculating the weakly connected components corresponding to each group, the target historical task instance indicated by the task instance mark corresponding to the group with the largest weakly connected component can be selected, and the task instance results in the target task instance can be reused during calculation.

[0093] Optionally, in this embodiment, if there are multiple directed graphs, for any directed graph, the directed graph is grouped according to the task instance labels of each node in the directed graph; and for the grouped directed graphs, the target historical task instance indicated by the task instance label corresponding to the group with the largest weakly connected component is selected as the target historical task instance corresponding to the directed graph. Thus, when there are multiple directed graphs, the target historical task instance corresponding to each directed graph can be determined separately.

[0094] When reusing task instance results, for the nodes of the reusable task instance results in the directed graph, the results corresponding to the nodes in the target historical task instance are stored; for the nodes in the directed graph other than the nodes of the reusable task instance results, the offline calculations corresponding to the nodes are performed based on the reused task instance results.

[0095] For example, see Figure 2D Two directed graphs are shown. Some nodes of the directed graph on the left side of the figure reuse the task instance results of historical task instance 1, and other nodes re-execute the calculation process. The nodes of the directed graph on the right side of the figure reuse the task instance results of historical task instance 2.

[0096] Optionally, in this embodiment, there may be a situation where multiple offline calculations are executed synchronously. In this case, if the result of the reused task instance in the target task instance has not been calculated, the reuse is performed after the calculation of the task instance result is completed.

[0097] In addition, in any embodiment of the present application, the method may further include: adjusting the life cycle of the table where the task instance result is located according to the frequency of reuse of the task instance result in the historical task instance. Thus, for the task instance result that is reused more frequently, the life cycle of the result table where it is located can be increased so that it can be reused continuously in the future.

[0098] Specifically, if the frequency of reuse of the task instance result is greater than the preset frequency, the life cycle of the result table where the task instance result is located is adjusted to the maximum period; or, if the frequency of reuse of the task instance result is greater than the preset frequency, the life cycle of the result table where the task instance result is located is increased by a preset multiple.

[0099] By way of example, in this embodiment, the average change time of the reference table during the calculation process can be recorded, and the time can be determined as the maximum period; if it is determined that the result table needs to be adjusted, the reference table of the node corresponding to the result table can be determined, and the life cycle of the result table can be adjusted to the maximum period corresponding to the determined reference table.

[0100] In addition, if the maximum value is not determined, the life cycle can be extended to a preset multiple of the original life cycle, such as twice the original life cycle.

[0101] The solution provided by this embodiment is to obtain a directed graph corresponding to the offline computing logic for the target data, and group the directed graph according to the task instance tag of each node in the directed graph, wherein the task instance tag is used to identify the historical task instance where the node can reuse the task instance result, calculate at least one weakly connected component in the directed graph part corresponding to each group, select the target historical task instance indicated by the task instance tag corresponding to the group with the weakly connected component in the front column, so as to reuse the task instance result of the target historical task instance, thereby, the task instance result in the reused historical task instance can be determined based on the node, rather than determined based on the incremental data, so that the data processing method provided by this embodiment is not limited by the incremental data, more universal, and has a wider range of applications; in addition, in this embodiment, the task instance result in the reused historical task instance is determined based on the node, which can greatly increase the number of nodes for reusing the task instance result, and further reduce the required computing resources and storage resources for storing the computing results. The solution provided by this embodiment can be directly executed for the preset node (such as SQL query node) that can reuse the task instance result.

[0102] It is known from experiments that, compared with a solution that generally reuses results based on incremental data, the solution provided in this embodiment improves the average calculation time of the model by 14.8%.

[0103] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.) and a PC, etc.

[0104] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application, as shown in the figure, includes:

[0105] An acquisition module 301 is used to acquire a directed graph corresponding to an offline computing logic for target data;

[0106] A grouping module 302, configured to group the directed graph according to the task instance tags of each node in the directed graph, wherein the task instance tags are used to identify the historical task instances where the reusable task instance results of the nodes are located;

[0107] Determine model 303, which is used to calculate at least one weakly connected component in the directed graph part corresponding to each group, and select the target historical task instance indicated by the task instance mark corresponding to the group with the weakly connected component in the front column to reuse the task instance result of the target historical task instance.

[0108] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the data processing device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0109] Embodiment 4

[0110] Reference Figure 4 , shows a schematic diagram of the structure of an electronic device according to Embodiment 5 of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0111] like Figure 4 As shown, the electronic device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0112] in:

[0113] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0114] The communication interface 404 is used to communicate with other electronic devices or servers.

[0115] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above data processing method embodiment.

[0116] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0117] The processor 402 may be a processor CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0118] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0119] The program 410 can be specifically used to enable the processor 402 to perform the operations in the above method embodiment. The specific implementation of each step in the program 410 can refer to the corresponding descriptions in the corresponding steps and units in the above data processing method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0120] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any data processing method in the above-mentioned multiple method embodiments.

[0121] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0122] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the data processing method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the data processing method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the data processing method shown here.

[0123] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0124] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A data processing method, comprising: Obtain a directed graph corresponding to the offline computing logic for the target data; According to the task instance label of each node in the directed graph, the directed graph is grouped, each group of partial directed graphs includes at least one node with the same task instance label, and the nodes in each group of partial directed graphs reuse the task instance results in the same historical task instance, wherein the task instance label is used to identify the historical task instance where the node reusable task instance result is located; Calculate at least one weakly connected component in the directed graph portion corresponding to each group, select the target historical task instance indicated by the task instance mark corresponding to the group with the weakly connected component in the front column, so as to reuse the task instance result of the target historical task instance.

2. The method according to claim 1, wherein: The grouping of the directed graph according to the task instance label of each node in the directed graph includes: According to the task instance labels of each node in the directed graph, the nodes are divided into multiple groups, wherein nodes with multiple task instance labels are located in multiple groups; According to the directed edges between the nodes in the directed graph, the nodes in each group are connected to obtain a partial directed graph corresponding to the group.

3. The method according to claim 1, wherein: If there are multiple directed graphs, grouping the directed graphs according to the task instance labels of the nodes in the directed graphs includes: For any directed graph, grouping the directed graph according to the task instance label of each node in the directed graph; The target historical task instance indicated by the task instance mark corresponding to the group whose weakly connected component is located in the front column is selected, including: For the grouped directed graph, the target historical task instance indicated by the task instance mark corresponding to the group with the largest weakly connected component is selected as the target historical task instance corresponding one-to-one to the directed graph.

4. The method according to claim 1, wherein: The task instance results of the target historical task instance are reused by the following steps: For the nodes of the reusable task instance results in the directed graph, storing the results corresponding to the nodes in the target historical task instance; For nodes in the directed graph other than nodes of reusable task instance results, offline calculations corresponding to the nodes are performed based on the reused task instance results.

5. The method according to claim 4, wherein: The method further comprises: If the result of the reused task instance in the target historical task instance has not been calculated, the reuse will be performed after the calculation of the task instance result is completed.

6. The method according to claim 1, wherein: The method further comprises: According to the frequency of reuse of task instance results in historical task instances, the life cycle of the table where the task instance results are located is adjusted.

7. The method according to claim 1, wherein: The task instance label of each node in the directed graph is determined by the following steps: For a directed graph corresponding to the offline computing logic of the target data, a current task instance corresponding to the directed graph is generated according to the target data, wherein the task instance includes current subtask instances corresponding one-to-one to the nodes of the directed graph; According to the current subtask instance corresponding to the node of the reusable task instance result, determining the reusable task instance result of the current subtask instance from the historical subtask instances corresponding to the node; According to the determined historical task instance where the reusable task instance result is located, a task instance tag is added to the node.

8. The method according to claim 7, wherein: Before determining the reusable task instance result of the current subtask instance from the historical subtask instances corresponding to the node according to the current subtask instance corresponding to the node of the reusable task instance result, the method further includes: Determine the idempotent nodes as nodes that can reuse the results of task instances.

9. The method according to claim 7, wherein: The step of determining the reusable task instance result of the current subtask instance from the historical subtask instances corresponding to the node according to the current subtask instance corresponding to the node of the reusable task instance result includes: According to the current subtask instance corresponding to the node of the reusable task instance result, determine the historical subtask instance with the same instance parameters as the current subtask instance and the same reference table from the historical subtask instances corresponding to the node, and determine its result as the reusable task instance result.

10. The method according to claim 7, wherein: The method further comprises: For nodes with upstream and downstream relationships in a directed graph, if the task instance tag of the downstream node does not correspond to the upstream node of the downstream node, the task instance tag corresponding to the downstream node is deleted.

11. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

12. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the data processing method according to any one of claims 1 to 10.

Images