Calculation process processing method and device

Abstract

The embodiment of the invention provides a processing method and device for a calculation process, and the method comprises the steps: obtaining a relation graph corresponding to the calculation process involving offline calculation and online calculation, enabling each node in the relation graph to represent each first operator of an offline type and each second operator of an online type involved in the calculation process, and enabling each first operator of the offline type and each second operator of the online type to represent each second operator of the online type; each directed edge represents a dependency relationship between the connected nodes; according to the operator type corresponding to each node in the relational graph, segmenting the relational graph to obtain a first sub-graph and a second sub-graph which are associated with each other; the incidence relation between the first sub-graph and the second sub-graph is recorded, nodes in the first sub-graph correspond to the first operator, and nodes in the second sub-graph correspond to the second operator; and based on the first sub-graph, the second sub-graph and the association relationship, running the calculation process to realize more convenient arrangement and deployment of the hybrid calculation process involving offline calculation and online calculation, thereby improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more particularly to a method and device for processing a computing process. Background Art

[0002] With the rapid development of the Internet, various big data computing platforms have emerged to provide data processing services for various enterprises and units. In order to better provide high-quality data processing services to various enterprises and units, it is crucial to provide more convenient orchestration and deployment solutions for the business computing processes of various enterprises and units. Summary of the invention

[0003] One or more embodiments of the present specification provide a method and device for processing a computing process to achieve more convenient orchestration and deployment of hybrid computing processes involving offline computing and online computing, thereby improving user experience.

[0004] According to a first aspect, a method for processing a computing process is provided, wherein the method comprises:

[0005] Obtain a relationship graph corresponding to the computing process, wherein each node in the relationship graph represents each operator involved in the computing process, each directed edge represents a dependency relationship between connected nodes, and the operator includes an offline first operator and an online second operator;

[0006] According to the operator type corresponding to each node in the relationship graph, the relationship graph is divided to obtain a first subgraph and a second subgraph associated with each other; and the association relationship between the first subgraph and the second subgraph is recorded, wherein the nodes in the first subgraph all correspond to the first operator, and the nodes in the second subgraph all correspond to the second operator;

[0007] The calculation process is executed based on the first subgraph, the second subgraph, and the association relationship.

[0008] According to a second aspect, a computing process processing device is provided, wherein the device comprises:

[0009] An acquisition module is configured to acquire a relationship graph corresponding to a computing process, wherein each node in the relationship graph represents each operator involved in the computing process, each directed edge represents a dependency relationship between connected nodes, and the operator includes an offline first operator and an online second operator;

[0010] A segmentation recording module is configured to segment the relationship graph according to the operator type corresponding to each node in the relationship graph to obtain an associated first subgraph and a second subgraph; and record the association relationship between the first subgraph and the second subgraph, wherein the nodes in the first subgraph all correspond to the first operator, and the nodes in the second subgraph all correspond to the second operator;

[0011] An operation module is configured to execute the calculation process based on the first subgraph, the second subgraph, and the association relationship.

[0012] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0013] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0014] According to the computing process processing method and device provided in the embodiments of this specification, a relationship graph corresponding to the computing process involving offline computing and online computing is obtained, wherein each node in the relationship graph respectively represents each first operator of the offline type and each second operator of the online type involved in the computing process, and each directed edge respectively represents the dependency relationship between the connected nodes; according to the operator type corresponding to each node in the relationship graph, the relationship graph is divided to obtain the associated first subgraph and second subgraph; the association relationship between the first subgraph and the second subgraph is recorded, wherein the nodes in the first subgraph all correspond to the first operator, and the nodes in the second subgraph all correspond to the second operator; based on the first subgraph, the second subgraph, and the association relationship, the computing process is run. The above process, based on the relationship diagram of the computing process involving offline computing and online computing, realizes the unified management of the offline computing sub-processes and online computing sub-processes involved in a computing process, and realizes the unified orchestration of this type of hybrid computing process. During the deployment process, based on the operator type corresponding to each node in the relationship diagram, the relationship diagram is segmented, and the association relationship between the first sub-graph and the second sub-graph obtained by the segmentation is recorded, so as to realize the automatic association of the association relationship between the offline computing sub-process and the online computing sub-process in the computing process, thereby reducing the complexity of the orchestration and deployment of the hybrid computing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in this specification;

[0017] Figure 2 A flowchart of a method for processing a calculation process provided in an embodiment;

[0018] Figure 3A A schematic diagram of a relationship diagram corresponding to the calculation process provided in the embodiment;

[0019] Figure 3B for Figure 3A A schematic diagram of the segmentation results of the provided relationship graph;

[0020] Figure 4 A schematic block diagram of a computing process processing device provided in an embodiment. DETAILED DESCRIPTION

[0021] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0022] The embodiments of this specification disclose a method and device for processing a computing process. The application scenarios and technical concepts of the method are first introduced as follows:

[0023] As mentioned above, with the rapid development of the Internet, various big data computing platforms have emerged to provide data processing services for various enterprises and units. In order to better provide high-quality data processing services to various enterprises and units, it is crucial to provide more traversal orchestration and deployment solutions for the business computing processes of various enterprises and units.

[0024] Considering that some real-time computing processes (i.e., online computing processes) often use some preprocessed data, these preprocessed data are often generated through offline computing. Accordingly, in the computing processes corresponding to the business of some enterprises and units, offline computing and online computing are often involved. For example, various computing processes based on neural network models often require offline training of the neural network model first (the process of training the neural network model is an offline computing sub-process), and then use the trained neural network model to provide online reasoning computing services (the process of providing online reasoning computing services is an online computing sub-process).

[0025] At present, in the process of orchestration and deployment of computing processes involving offline computing and online computing, the computing processes are generally divided into two types of processes for separate orchestration, namely, offline computing sub-processes and online computing sub-processes; after completing the orchestration of the offline computing sub-processes, each offline computing sub-process is deployed to run each offline computing sub-process to obtain its result data; then, the result data of each offline computing sub-process is imported into the online computing sub-process after orchestration and deployment to provide online computing services.

[0026] In the above process, in the process of importing the result data of each offline computing sub-process into the online computing sub-process after orchestration and deployment, the user is generally required to manually configure the result data of each offline computing sub-process into the online computing sub-process in the form of parameters. In this process, the user is required to manually specify and record which computing link (or step, functional unit) in the online computing sub-process the result data of each offline computing sub-process needs to be imported into. This results in a high degree of user operation complexity in the current orchestration and deployment process of the computing process.

[0027] In view of this, the inventor proposes a method for processing computing processes, which can realize the automatic association of the relationship between offline computing sub-processes and online computing sub-processes involved in a computing process, thereby reducing the operational complexity of users in the process of orchestration and deployment of such hybrid computing processes.

[0028] Figure 1 A schematic diagram of an implementation scenario according to an embodiment disclosed in this specification is shown. In this implementation scenario, a relationship graph corresponding to a computing process is obtained, wherein each node in the relationship graph represents each operator involved in the computing process, each directed edge represents the dependency relationship between the connected nodes, the computing process involves offline computing and online computing, and each operator corresponding to each node in the corresponding relationship graph includes a first operator of an offline type and a second operator of an online type. Figure 1 As shown, the relationship graph can be compiled by the user according to the various computing links in the computing process of his business and the dependencies between the computing links. After obtaining the relationship graph, the relationship graph is divided according to the operator type corresponding to each node in the relationship graph to obtain the associated first subgraph and second subgraph; and the association relationship between the first subgraph and the second subgraph is recorded, wherein the nodes in the first subgraph all correspond to the first operator, that is, the first subgraph is the subgraph corresponding to the offline computing sub-process in the computing process, and the nodes in the second subgraph all correspond to the second operator, that is, the second subgraph is the subgraph corresponding to the online computing sub-process in the computing process; then, based on the first subgraph, the second subgraph, and the association relationship, the computing process is run. Exemplarily, the obtained first subgraph can be one or more, and the obtained second subgraph can be one or more.

[0029] In the above process, based on the relationship diagram of the computing process involving offline computing and online computing, unified management of offline computing sub-processes and online computing sub-processes involved in a computing process is achieved, and unified orchestration of this type of hybrid computing process is achieved. During the deployment process, based on the operator type corresponding to each node in the relationship diagram, the relationship diagram is segmented, and the association relationship between the first sub-graph and the second sub-graph obtained by the segmentation is recorded, so as to realize the automatic association of the association relationship between the offline computing sub-process and the online computing sub-process in the computing process, thereby reducing the complexity of the orchestration and deployment of the hybrid computing process.

[0030] The processing method of the calculation flow provided in this specification is described in detail below in conjunction with specific embodiments.

[0031] Figure 2 The flowchart of a method for processing a computing process in one embodiment of the present specification is shown. The method is executed by an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. In the processing of the computing process, such as Figure 2 As shown, the method includes the following steps S210-S250:

[0032] In step S210, a relationship graph corresponding to the computation process is obtained, wherein each node in the relationship graph represents each operator involved in the computation process, each directed edge represents a dependency relationship between connected nodes, and the operator includes an offline first operator and an online second operator.

[0033] In some possible examples, the target program for implementing the processing flow of the computing flow provided in the embodiments of this specification may exist in the form of an independent client, and accordingly, the client may be bound to a specified big data computing platform so as to call the offline computing services and online computing services provided by the specified big data computing platform to run the computing flow. The specified big data computing platform may be any big data computing platform that can be used to provide data computing services to various parties (e.g., enterprises and units). In some cases, the online computing service may also be referred to as a real-time computing service.

[0034] In some other possible examples, the target program for implementing the processing flow of the computing flow provided in the embodiments of this specification may also exist in the form of a plug-in for a specified big data platform, so that offline computing services and online computing services provided by the specified big data computing platform can be directly called.

[0035] The aforementioned target program may be installed in an electronic device, and the corresponding electronic device executes the processing method of the computing process by running the target program.

[0036] In some examples, the electronic device can obtain the relationship diagram corresponding to the calculation process (hereinafter referred to as calculation process X) uploaded by the user to be processed, so as to execute the subsequent process for the relationship diagram. In another example, the electronic device (through the target program) provides a relationship diagram arrangement function for the user to construct the corresponding relationship diagram according to the various calculation links of its calculation process X. Exemplarily, the electronic device can provide an arrangement page with a relationship diagram, and the arrangement page can display icons corresponding to various operators for the user to choose to construct a relationship diagram. Specifically, the user can select the corresponding operator (i.e., each operator of the functional unit required to implement each calculation link) on the arrangement page according to each calculation link of the calculation process X that needs to be arranged, and construct the dependency relationship between the selected operators according to the dependency relationship between the calculation links to construct the corresponding relationship diagram, so that the electronic device obtains the relationship diagram corresponding to the calculation process.

[0037] Specifically, the relationship graph represents each operator corresponding to each computing link of the computing process through each node, and represents the dependency relationship between the nodes, that is, the dependency relationship between the operators corresponding to the nodes, through the directed edges between the nodes. For example, directed edge e1 connects node 1 and node 2, and points from node 1 to node 2. The dependency relationship between node 1 and node 2 represented by it is: after the execution of the operator corresponding to node 1 is completed, the operator corresponding to node 2 is executed based on the execution result of the operator corresponding to node 1.

[0038] Among them, an operator can refer to a packaged functional unit. This functional unit has several input and output ports, which can obtain the expected output content according to the input content and its configured parameters through the pre-packaged data analysis and calculation process. The user does not need to perceive its internal processing process. In some cases, operators are classified from the perspective of relevant attributes of the calculation (offline calculation or online calculation). Operators can include operators involved in offline calculations (hereinafter referred to as offline type operators) and operators involved in online calculations (hereinafter referred to as online type operators).

[0039] In some other cases, from the perspective of the functions implemented by the operators, the operators can also be classified as follows: data I / O operators, feature engineering operators, learning operators and statistical operators. For example, data I / O operators may include operators that can obtain data from a specified data source, store the obtained data in a specified location, and / or convert the format of the data into a storable or transmittable format. Feature engineering operators may include operators that can preprocess data (such as data screening, cleaning, etc.), extract data to extract useful features, process features to obtain new features, and screen and convert features. Machine learning operators may include operators that can calculate and predict data using trained models, and / or evaluate the performance of the model, prediction accuracy and other indicators. Statistical operators may include operators that can perform statistics on data, and / or analyze the distribution characteristics of data, etc. It can be understood that for offline computing or online computing, the functions of the operators involved may be diverse. For example, both offline computing and online computing may involve data I / O operators.

[0040] In step S210, the calculation process X acquired by the electronic device involves offline calculation and online calculation, and accordingly, the calculation process X includes an offline calculation sub-process and an online calculation sub-process. The operators corresponding to each node in the corresponding relationship diagram include a first operator of an offline type involved in the offline calculation sub-process and a second operator of an online type involved in the online calculation sub-process.

[0041] In some specific examples, the relationship graph is a directed acyclic graph. In order to ensure the accurate execution of the processing flow, after the electronic device obtains the relationship graph corresponding to the calculation process, it can determine whether there is a loop in the relationship graph. If it is determined that there is a loop (for example, there is no node with a corresponding in-degree value of 0 in the relationship graph), the subsequent processing flow can be discontinued, and a prompt message can be output to prompt the user that there is a loop problem in the relationship graph; if it is determined that there is no loop, the subsequent processing flow can be continued. Among them, the in-degree value corresponding to a node can refer to the number of nodes that are its predecessor nodes.

[0042] Next, in step S220, the relationship graph is segmented according to the operator type corresponding to each node in the aforementioned relationship graph to obtain an associated first subgraph and a second subgraph; the association relationship between the first subgraph and the second subgraph is recorded, wherein the nodes in the aforementioned first subgraph all correspond to the first operator, and the nodes in the aforementioned second subgraph all correspond to the second operator.

[0043] In this step, the electronic device can traverse each node of the acquired relationship graph, determine the operator type corresponding to each node, and split the relationship graph based on the operator type corresponding to each node, so as to split the nodes that have dependencies and correspond to the same operator type into the same subgraph, and obtain the associated first subgraph and second subgraph, wherein the nodes in the first subgraph all correspond to the first operator, that is, the first subgraph is a graph corresponding to an offline computing subprocess in the computing process; the nodes in the second subgraph all correspond to the second operator, that is, the second subgraph is a graph corresponding to an online computing subprocess in the computing process.

[0044] It can be understood that the first subgraph obtained by splitting the relationship graph can be one or more, and the second subgraph can be one or more. The embodiments of this specification do not limit the number of the first subgraph and the second subgraph obtained by splitting the relationship graph. The number of the associated first subgraph and the second subgraph obtained is related to each calculation link of the calculation process corresponding to the relationship graph.

[0045] In some possible examples, in step S220, according to the operator type corresponding to each node in the aforementioned relationship graph, the process of segmenting the relationship graph to obtain the associated first subgraph and second subgraph may include steps 11-12:

[0046] In step 11, in the aforementioned relationship graph, according to the direction of the directed edge, at least one pair of target nodes corresponding to different operator types and being predecessor nodes and successor nodes to each other are found, and the at least one pair of target nodes is used to determine the split position in the aforementioned relationship graph. In this step, the electronic device can start from the node without an incoming edge in the aforementioned relationship graph, and according to the direction of the directed edge in the relationship graph, find at least one pair of target nodes corresponding to different operator types and being predecessor nodes and successor nodes to each other. Figure 3A The figure shows an example of a relationship diagram corresponding to a computational process, where hollow nodes are nodes corresponding to the second operator, and shaded nodes are nodes corresponding to the first operator. The numbers in the nodes represent different nodes corresponding to the various computational links in the computational process, and there are no other limiting meanings for the nodes. For example, for node 3 and node 6, the two are connected by a directed edge, and node 3 points to node 6. Accordingly, node 3 is the predecessor node of node 6, and node 6 is the successor node of node 3. Figure 3A In , the target node pairs include 3 pairs, namely node 1 and node 8, node 7 and node 8, and node 7 and node 9. Figure 3A Three split positions are determined in the relationship graph shown. It can be understood that one target node pair corresponds to one split position, and the relationship graph is split at the split position.

[0047] Then, in step 12, based on each segmentation position, the associated first subgraph and second subgraph are obtained from the aforementioned relationship graph. In this step, the electronic device can determine at least one segmentation position in the relationship graph based on the at least one pair of target nodes found, and the electronic device can obtain the associated first subgraph and second subgraph from the relationship graph based on each segmentation position.

[0048] In some specific examples, step 11 may include steps 111-113:

[0049] In step 111, any node whose corresponding in-degree value is 0 is determined from the nodes in the aforementioned relationship graph that have not been traversed, as the first node.

[0050] In step 112, starting from the first node, the nodes in the aforementioned relationship graph that have not been traversed are traversed according to the directions of the first directed edges connected to the first node.

[0051] In step 113, in the direction of each first directed edge, if a second node whose corresponding operator type has changed is traversed, the second node and its preceding node are determined to be a target node pair.

[0052] In some implementations, after the electronic device obtains the relationship graph, it can first determine the in-degree value corresponding to each node based on the dependency relationship between the nodes in the relationship graph, wherein the in-degree value corresponding to the node is equal to the number of the node's predecessor nodes. Then, from the nodes that have not been traversed in the relationship graph, determine any node whose current corresponding in-degree value is 0 as the first node; and starting from the first node, traverse only the nodes that are currently determined to be untraversed in the relationship graph according to the directions of the directed edges connected to the first node (hereinafter referred to as the first directed edges); in the direction of each first directed edge, if the corresponding second node whose operator type has changed is traversed, determine that the second node and its predecessor node are a pair of target nodes, that is, determine a split position.

[0053] For example, Figure 3AAs shown, in the first traversal, all nodes in the relationship graph are nodes that have not been traversed, wherein the in-degree values ​​corresponding to nodes 1, 2, 3, and 4 are all 0, and any one of the four nodes can be determined as the first node, for example, node 2 is determined as the first node, wherein node 2 is connected to a first directed edge, and the operator type corresponding to node 2 is an offline type. In the relationship graph, starting from node 2, according to the direction of the first directed edge connected to node 2, the nodes in the relationship graph that are currently determined to have not been traversed are traversed, and traversed to node 5, and the operator type corresponding to node 5 is an offline type (the operator type has not changed); then, continue to traverse the nodes in the relationship graph that are currently determined to have not been traversed according to the direction of the directed edge connected to node 5 (that is, the direction of the first directed edge connected to node 2), and traverse to node 7, and the operator type corresponding to node 7 is an offline type (the corresponding operator type has not changed).

[0054] Continue to traverse the nodes in the relationship graph that are currently determined to be untraversed according to the direction of the directed edge connected to node 7 (that is, the direction of the first directed edge connected to node 2), where there are two directed edges connected to node 7, one of which points from node 7 to node 8, and the other from node 7 to node 9; first traverse to node 8, the operator type corresponding to node 8 is the online type (the corresponding operator type has changed), that is, node 8 is the second node (relative to node 2), and node 8 and its predecessor node, node 7, are determined as a pair of target node pairs; then traverse to node 9, the operator type corresponding to node 9 is the online type (the corresponding operator type has changed), that is, node 9 is the second node (relative to node 2), and node 9 and its predecessor node, node 7, are determined as a pair of target node pairs. Each pair of target node pairs corresponds to a split position.

[0055] In some possible examples, in order to ensure the accuracy of the segmentation result of the relationship graph, after determining in step 113 that the second node and its predecessor node are a pair of target nodes, step 114 is also included: in step 114, the in-degree value corresponding to the second node is reduced by 1, and the second node is determined to have not been traversed. Continuing with the above example, after determining node 8 and its predecessor node, node 7, as a pair of target nodes, the in-degree value (2) corresponding to node 8 (the second node) is reduced by 1, and the current corresponding in-degree value of node 8 is 1; and node 8 is determined to have not been traversed. Similarly, for node 9, the in-degree value (2) corresponding to node 9 (the second node) is reduced by 1, and the current corresponding in-degree value of node 9 is 1; and node 9 is determined to have not been traversed.

[0056] In some possible examples, the electronic device may continue to return to execute steps 111-114, and so on, until there are no untraversed nodes in the relationship graph (that is, there are no nodes determined to be untraversed), so as to obtain all target node pairs in the relationship graph, that is, all segmentation positions. Figure 3A The relationship diagram shown is used as an example for explanation. At this time, there are nodes in the relationship diagram that are currently determined to be untraversed, and there are nodes (including node 1, node 3 and node 4) whose corresponding in-degree values ​​are currently 0. For example, node 1 is the first node, wherein the operator type corresponding to node 1 is an offline type; in the relationship diagram, starting from node 1, according to the direction of the first directed edge connected to node 1, the nodes currently determined to be untraversed in the relationship diagram are traversed, and in the direction of the first directed edge connected to node 1, node 8 is traversed to, and the operator type corresponding to node 8 is an online type (the corresponding operator type changes), that is, node 8 is the second node (relative to node 1), and node 8 and its predecessor node, node 1, are determined as a pair of target nodes. Afterwards, the in-degree value corresponding to node 8 (which is 1) is subtracted by 1, and the current corresponding in-degree value of node 8 is obtained to be 0; and node 8 is set to be untraversed. Afterwards, the electronic device continues to return to execute steps 111-114.

[0057] Afterwards, after all target node pairs are determined in the relationship graph, the associated first subgraph and second subgraph may be obtained from the relationship graph based on the respective segmentation positions corresponding to all target node pairs.

[0058] In some other implementations, if there are untraversed nodes in the relationship graph, but there is no node whose current corresponding indegree value is 0, that is, it is impossible to determine a node that can be used as the first node from the nodes that have not been traversed in the relationship graph, then it is considered that there is a loop in the relationship graph, and the subsequent processing flow may no longer be continued, and a prompt message may be output to prompt the user that there is a loop problem in the relationship graph.

[0059] In some specific examples, step 12 may include step 121: in step 121, based on the first node, and multiple connected nodes between the first node and the second node, and the operator type corresponding to the first node, a first subgraph or a second subgraph is constructed. In this step, for a first node, if there is a second node whose corresponding operator type changes in the direction of the first directed edge connected to the first node, the electronic device constructs the first subgraph or the second subgraph based on the first node, and multiple connected nodes between the first node and the second node, and the operator type corresponding to the first node after determining the corresponding second node. Wherein, if the operator type corresponding to the first node is an offline type, the first subgraph is constructed based on the first node, and multiple connected nodes between the first node and the second node; if the operator type corresponding to the first node is an online type, the second subgraph is constructed based on the first node, and multiple connected nodes between the first node and the second node.

[0060] For example, when node 2 is the first node, its corresponding operator type is an offline type, and its corresponding second node includes node 8 and node 9. A first subgraph can be constructed based on node 2, node 5, and node 7. When node 1 is the first node, its corresponding operator type is an offline type, and its corresponding second node includes node 8. No other nodes are included between node 1 and node 8, and a first subgraph is constructed based on node 1. When node 4 is the first node, its corresponding operator type is an online type. In one direction of the first directed edge connected to it, the corresponding second node is node 7, and there is no second node in the other direction of the first directed edge connected to it. At this time, a second subgraph can be constructed based on node 4, node 9, and node 11.

[0061] In some specific examples, in step 121, the process of constructing the first subgraph or the second subgraph may specifically include the following steps: if the predecessor node of the second node has been traversed in the traversal process starting from other first nodes, and the operator type corresponding to the other first node is the same as the operator type corresponding to the first node, then based on the first node, and multiple connected nodes between the first node and the second node, and other first nodes, and multiple connected nodes between the other first nodes and the predecessor node of the second node, the first subgraph or the second subgraph is constructed.

[0062] by Figure 3AAs shown, in the case where node 3 is the first node, the operator type corresponding to node 3 is an offline type, and the second node corresponding to node 3 includes node 9. At this time, the predecessor node of node 9, that is, node 7, has been traversed in the traversal process starting from node 2 (that is, the other first node), and the operator type corresponding to node 3 (the first node at this time) is the same as the operator type corresponding to node 2 (that is, the other first node) (for example, both are offline types), then based on node 3 (the first node at this time), and node 6 connected between node 3 (the first node at this time) and node 9 (the second node), and node 2 (that is, the other first node), and node 5 connected between node 2 and node 9's predecessor node, that is, node 7, a first subgraph is constructed, wherein the first subgraph includes node 2, node 3, node 5, node 6 and node 7.

[0063] When node 1 is taken as the first node, node 1 has no other nodes with dependencies except node 8, and node 1 can be taken as a first subgraph.

[0064] In some possible examples, in the process of traversing the nodes determined as not traversed in the relationship graph, it is inevitable that there is no second node whose corresponding operator type has changed in the pointing direction of the first directed edge connected to the determined first node. In this case, step 12 may also include step 121: in the pointing direction of each first directed edge, if the second node whose corresponding operator type has changed has not been traversed, construct a first subgraph or a second subgraph based on the first node and all its nodes in the pointing direction of each first directed edge, and the operator type corresponding to the first node, so as to obtain the associated first subgraph and second subgraph.

[0065] by Figure 3A Taking the relationship graph shown as an example, after determining the target node pair of node 7 and node 9, if there are nodes 4, 9 and 11 that are determined as nodes that have not been traversed in the relationship graph, node 4 can be determined as the first node from the nodes that have not been traversed in the relationship graph, and starting from node 4, the untraversed nodes in the relationship graph are traversed according to the directions of the first directed edges connected to them, and a second node whose corresponding operator type has not changed in the direction of the first directed edge of node 4 is determined, then a second subgraph is constructed based on the node and all its nodes in the direction of the first directed edge, and the operator type corresponding to node 4 (which is an online type).

[0066] Based on this Figure 3A As shown in the relationship diagram, we can get Figure 3BShown are two first subgraphs (one of which includes node 1, and the other includes node 2, node 3, node 5, node 6, and node 7), and two second subgraphs (one of which includes node 8 and node 10, and the other includes node 4, node 9, and node 11).

[0067] Considering that for any pair of target nodes, the execution of the operator corresponding to the successor node in the target node pair depends on the execution result of the operator corresponding to the predecessor node in the target node pair (generally an offline type operator), in order to ensure the normal execution of the online calculation sub-process corresponding to the second subgraph, after obtaining the associated first subgraph and second subgraph, it is also necessary to record the association relationship between the first subgraph and the second subgraph based on the split position between the first subgraph and the second subgraph (that is, the aforementioned split position). In some possible examples, in step S220, the process of recording the association relationship between the first subgraph and the second subgraph may include steps 21-22:

[0068] In step 21, a data identifier is generated for the data corresponding to the split position between the first subgraph and the second subgraph, wherein the data corresponding to the split position is the output of the preceding node in the corresponding target node pair. Figure 3B As shown, for the first child Figure 1 and the second son Figure 1 The data corresponding to the split position between nodes 1 and 8 (i.e., the split position between nodes 1 and 8) is generated as a data identifier id: A. Figure 2 and the second son Figure 1 The data corresponding to the split position between nodes 7 and 8 (i.e., the split position between nodes 7 and 8) generates a data identifier id: B. Figure 2 and the second son Figure 2 The data corresponding to the split position between nodes 7 and 9 (i.e., the split position between nodes 7 and 9) generates a data identifier id: C.

[0069] Next, in step 22, the data identifier of the data corresponding to the split position and the association relationship with the successor node in the target node pair corresponding to each split position are recorded. In this step, the electronic device needs to record the data identifier of the data corresponding to the split position and the association relationship with the successor node in the target node pair corresponding to each split position, so as to record the association relationship between the first subgraph and the second subgraph, and realize the automatic association between the offline calculation sub-process and the online calculation sub-process involved in the calculation process. For example, the second subgraph can be recorded. Figure 1 The input content corresponding to the node 8 includes the data corresponding to the data identifier id: A and the data corresponding to the data identifier id: B; record the second child Figure 2 The input content corresponding to node 9 includes data identifier id: data corresponding to C.

[0070] Then, in step S230, based on the first subgraph, the second subgraph, and the association relationship, the calculation process is run. In this step, the electronic device can deploy the first subgraph and the second subgraph respectively, and then first run the offline calculation sub-process corresponding to the first subgraph, and based on the association relationship between the first subgraph and the second subgraph, import the operation result data after the operation corresponding to the first subgraph into the second subgraph, and then run the online calculation sub-process corresponding to the second subgraph after the operation result data of the first subgraph is imported to realize the operation of the calculation process.

[0071] Specifically, in some possible examples, step S230 may include the following steps 31-32:

[0072] In step 31, the first subgraph is deployed, and the first operator corresponding to the node in the deployed first subgraph is run to obtain the first result data corresponding to the first subgraph. In this step, the electronic device can deploy the first subgraph to the offline computing sub-platform provided by the aforementioned designated big data computing platform to call the offline computing service provided by it, run the first operator corresponding to the node in the deployed first subgraph, and obtain the first result data corresponding to the first subgraph.

[0073] In step 32, the second subgraph is deployed, and the first result data is imported into the second subgraph based on the association relationship to provide real-time computing services. In this step, the electronic device can deploy the second subgraph to the online computing sub-platform provided by the aforementioned designated big data computing platform to call the online computing service provided by it to implement the operation of the online computing sub-process corresponding to the second subgraph. Then, after obtaining the first result data corresponding to the first subgraph, the electronic device imports the first result data into the second subgraph (i.e., imports it into the corresponding operator) based on the recorded association relationship to provide real-time computing services by importing the online computing sub-process corresponding to the second subgraph of the first result data.

[0074] In the above process, based on the relationship diagram of the computing process involving offline computing and online computing, unified management of offline computing sub-processes and online computing sub-processes involved in a computing process is achieved, and unified orchestration of this type of hybrid computing process is achieved. During the deployment process, based on the operator type corresponding to each node in the relationship diagram, the relationship diagram is segmented, and the association relationship between the first sub-graph and the second sub-graph obtained by the segmentation is recorded, so as to realize the automatic association of the association relationship between the offline computing sub-process and the online computing sub-process in the computing process, thereby reducing the complexity of the orchestration and deployment of the hybrid computing process.

[0075] In the above process, when the association relationship between the first subgraph and the second subgraph remains unchanged (i.e., the association relationship between the offline computing subprocess and the online computing subprocess remains unchanged, i.e., the data identifiers corresponding to the successor nodes in the target node pairs corresponding to each segmentation position remain unchanged), when the operation process of the offline computing subprocess in the computing process changes, the online computing subprocess side is imperceptible, and the online computing subprocess side can complete the data update imperceptibly (for example, the update of the parameters of the model involved in the offline computing subprocess is imperceptible). Similarly, when the association relationship between the first subgraph and the second subgraph remains unchanged, when the operation process of the online computing subprocess in the computing process changes, the offline computing subprocess side is imperceptible.

[0076] In some possible examples, in the process of running the online computing subprocess corresponding to the second subgraph to provide real-time computing services, it is inevitable that the result data of the offline computing subprocess corresponding to the first subgraph needs to be updated. For example, in various computing processes based on neural network models, the parameters of the neural network model need to be periodically updated. Accordingly, the processing method of the computing process may further include steps 41-42:

[0077] In step 41, if it is determined that the data of the offline computing sub-process needs to be updated, the deployed first sub-graph is triggered to obtain the second result data corresponding to the first sub-graph, wherein the offline computing sub-process belongs to the aforementioned computing process. In this step, if the electronic device determines that the data of the offline computing sub-process needs to be updated, the deployed corresponding first sub-graph is triggered to run the offline computing sub-process corresponding to the first sub-graph to obtain the second result data corresponding to the first sub-graph.

[0078] Next, in step 42, based on the aforementioned association relationship, the second result data is imported into the second subgraph to replace the first result data. In this step, the electronic device imports the second result data into the corresponding operator of the online calculation subprocess corresponding to the second subgraph based on the aforementioned association relationship to replace the first result data therein, so as to update the parameters of the corresponding operator in the online calculation subprocess.

[0079] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] Corresponding to the above method embodiment, the present specification embodiment provides a processing device 400 for calculating a process, and its schematic block diagram is as follows: Figure 4 As shown, the device comprises:

[0081] The acquisition module 410 is configured to acquire a relationship graph corresponding to the computing process, wherein each node in the relationship graph represents each operator involved in the computing process, each directed edge represents a dependency relationship between connected nodes, and the operator includes an offline type first operator and an online type second operator;

[0082] The segmentation recording module 420 is configured to segment the relationship graph according to the operator type corresponding to each node in the relationship graph to obtain an associated first subgraph and a second subgraph; record the association relationship between the first subgraph and the second subgraph, wherein the nodes in the first subgraph all correspond to the first operator, and the nodes in the second subgraph all correspond to the second operator;

[0083] The running module 430 is configured to run the calculation process based on the first subgraph, the second subgraph, and the association relationship.

[0084] In some possible examples, the running module 430 is specifically configured to deploy the first subgraph, run the first operator corresponding to the node in the deployed first subgraph, and obtain the first result data corresponding to the first subgraph;

[0085] The second subgraph is deployed, and the first result data is imported into the second subgraph based on the association relationship to provide a real-time computing service.

[0086] In some possible examples, the relationship graph is a directed acyclic graph.

[0087] In some possible examples, the record segmentation module 420 includes:

[0088] A search unit (not shown in the figure) is configured to search for at least one pair of target nodes that have different operator types and are predecessor nodes and successor nodes of each other in the relationship graph according to the directions of the directed edges therein, wherein the at least one pair of target nodes is used to determine a split position in the relationship graph;

[0089] The obtaining unit (not shown in the figure) is configured to obtain the associated first sub-graph and second sub-graph from the relationship graph based on each segmentation position.

[0090] In some possible examples, the search unit is specifically configured to determine any node whose current corresponding in-degree value is 0 from the nodes that have not been traversed in the relationship graph as the first node;

[0091] Starting from the first node, traversing the nodes in the relationship graph that have not been traversed according to the directions of the first directed edges connected to the first node;

[0092] In the direction of each first directed edge, if a second node whose corresponding operator type has changed is traversed, it is determined that the second node and its preceding node are a pair of target nodes.

[0093] Some possible examples include:

[0094] A determination unit (not shown in the figure) is configured to, after determining that the second node and its predecessor node are a pair of target nodes, reduce the in-degree value corresponding to the second node by 1 and determine the second node as not having been traversed.

[0095] In some possible examples, the obtaining unit includes:

[0096] A first construction sub-unit (not shown in the figure) is configured to construct a first subgraph or a second subgraph based on the first node, a plurality of connected nodes between the first node and the second node in the direction of the first directed edge thereof, and an operator type corresponding to the first node, so as to obtain associated first subgraphs and second subgraphs.

[0097] In some possible examples, the first construction subunit is specifically configured to construct the first subgraph based on the first node, and multiple connected nodes between the first node and the second node, and the other first nodes, and multiple connected nodes between the other first nodes and the predecessor nodes of the second node, if the predecessor node of the second node has been traversed in the traversal process starting from other first nodes, and the operator type corresponding to the other first nodes is the same as the operator type corresponding to the first node.

[0098] In some possible examples, the obtaining unit further includes:

[0099] The second construction sub-unit (not shown in the figure) is configured to construct the first subgraph or the second subgraph based on the first node and all nodes thereof in the direction of each first directed edge, and the operator type corresponding to the first node, so as to obtain associated first subgraphs and second subgraphs, if the second node whose corresponding operator type has changed has not been traversed in the direction of each first directed edge.

[0100] In some possible examples, the segmentation record module 420 is specifically configured as follows:

[0101] Generate a data identifier for data corresponding to a split position between the first subgraph and the second subgraph, wherein the data corresponding to the split position is an output of a preceding node in the corresponding target node pair;

[0102] Record the data identifier of the data corresponding to the split position and the association relationship with the successor node in the target node pair corresponding to each split position.

[0103] Some possible examples include:

[0104] A trigger module (not shown in the figure) is configured to trigger the deployed first subgraph to obtain second result data corresponding to the first subgraph if it is determined that data update of the offline computing subprocess is required, wherein the offline computing subprocess belongs to the computing process;

[0105] An import module (not shown in the figure) is configured to import the second result data into the second subgraph based on the association relationship to replace the first result data.

[0106] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0107] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the processing method of the calculation process provided in the present specification.

[0108] An embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the processing method of the computing flow provided in the present specification is implemented.

[0109] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0110] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0111] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for processing a computational process, wherein: The method comprises: Obtain a relationship graph corresponding to the computing process, wherein each node in the relationship graph represents each operator involved in the computing process, each directed edge represents a dependency relationship between connected nodes, and the operator includes an offline first operator and an online second operator; According to the operator type corresponding to each node in the relationship graph, the relationship graph is divided to obtain a first subgraph and a second subgraph associated with each other; and the association relationship between the first subgraph and the second subgraph is recorded, wherein the nodes in the first subgraph all correspond to the first operator, and the nodes in the second subgraph all correspond to the second operator; The calculation process is executed based on the first subgraph, the second subgraph, and the association relationship.

2. The method of claim 1, wherein: The step of running the calculation process based on the first subgraph, the second subgraph, and the association relationship includes: Deploy the first subgraph, run the first operator corresponding to the node in the deployed first subgraph, and obtain first result data corresponding to the first subgraph; The second subgraph is deployed, and the first result data is imported into the second subgraph based on the association relationship to provide a real-time computing service.

3. The method of claim 1, wherein: The relationship graph is a directed acyclic graph.

4. The method of claim 1, wherein: The step of segmenting the relationship graph according to the operator type corresponding to each node in the relationship graph to obtain the associated first subgraph and second subgraph includes: In the relationship graph, according to the directions of the directed edges therein, at least one pair of target nodes corresponding to different operator types and being predecessor nodes and successor nodes of each other are found, and the at least one pair of target nodes is used to determine the split position in the relationship graph; Based on each segmentation position, an associated first subgraph and a second subgraph are obtained from the relationship graph.

5. The method of claim 4, wherein: The finding of at least one pair of target nodes that have different operator types and are predecessor nodes and successor nodes to each other includes: From the nodes in the relationship graph that have not been traversed, determine any node whose corresponding in-degree value is 0 as the first node; Starting from the first node, traversing the nodes in the relationship graph that have not been traversed according to the directions of the first directed edges connected to the first node; In the direction of each first directed edge, if a second node whose corresponding operator type has changed is traversed, it is determined that the second node and its preceding node are a pair of target nodes.

6. The method according to claim 5, after determining that the second node and its predecessor node are a pair of target nodes, further comprising: The in-degree value corresponding to the second node is reduced by 1, and the second node is determined as not having been traversed.

7. The method of claim 6, wherein: The obtaining the associated first subgraph and second subgraph from the relationship graph based on each segmentation position includes: Based on the first node, and multiple connected nodes between the first node and the second node in the direction of the first directed edge thereof, and the operator type corresponding to the first node, a first subgraph or a second subgraph is constructed to obtain associated first subgraph and second subgraph.

8. The method of claim 7, wherein: The constructing of the first subgraph or the second subgraph comprises: If the predecessor node of the second node has been traversed during the traversal process starting from other first nodes, and the operator type corresponding to the other first nodes is the same as the operator type corresponding to the first node, then the first subgraph or the second subgraph is constructed based on the first node, and multiple connected nodes between the first node and the second node, and the other first nodes, and multiple connected nodes between the other first nodes and the predecessor nodes of the second node.

9. The method according to claim 7, wherein obtaining the associated first subgraph and second subgraph from the relationship graph based on each segmentation position further comprises: In the direction of each first directed edge, if the second node whose corresponding operator type has changed has not been traversed, a first subgraph or a second subgraph is constructed based on the first node and all nodes thereof in the direction of each first directed edge, and the operator type corresponding to the first node, so as to obtain the associated first subgraph and second subgraph.

10. The method of claim 4, wherein: The recording of the association relationship between each first sub-graph and the second sub-graph includes: Generate a data identifier for data corresponding to a split position between the first subgraph and the second subgraph, wherein the data corresponding to the split position is an output of a preceding node in the corresponding target node pair; Record the data identifier of the data corresponding to the split position and the association relationship with the successor node in the target node pair corresponding to each split position.

11. The method according to any one of claims 1 to 10, further comprising: If it is determined that data update of the offline computing sub-process is required, triggering the deployed first subgraph to obtain second result data corresponding to the first subgraph, wherein the offline computing sub-process belongs to the computing process; Based on the association relationship, the second result data is imported into the second subgraph to replace the first result data.

12. A processing device for a computing process, wherein: The device comprises: An acquisition module is configured to acquire a relationship graph corresponding to a computing process, wherein each node in the relationship graph represents each operator involved in the computing process, each directed edge represents a dependency relationship between connected nodes, and the operator includes an offline first operator and an online second operator; A segmentation recording module is configured to segment the relationship graph according to the operator type corresponding to each node in the relationship graph to obtain an associated first subgraph and a second subgraph; and record the association relationship between the first subgraph and the second subgraph, wherein the nodes in the first subgraph all correspond to the first operator, and the nodes in the second subgraph all correspond to the second operator; An operation module is configured to execute the calculation process based on the first subgraph, the second subgraph, and the association relationship.

13. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.

Images