A processing system and method based on cross-network graphic data structure fusion

Abstract

The present invention provides a processing system and method based on cross-network graph data structure fusion, which relates to the field of computer model technology. The system includes: a first acquisition module, which is used to acquire a set of tasks to be executed by a set of computing units; a graph data structure module, which is used to determine the graph data structure of the tasks to be executed; a fusion node module, which is used to determine the fusion node; a target graph data structure module, which is used to determine the target graph data structure; a fusion graph data structure module, which is used to obtain the fusion graph data structure; a second acquisition module, which is used to acquire the computing resource demand and predict the computing time; an allocation module, which is used to allocate computing units; and an execution module, which is used to process the fusion graph data structure. According to the present invention, multiple graph data structures can be simplified, computing units can be allocated to nodes with similar characteristics, and corresponding computing resources can be provided, so that the allocation of computing resources can be optimized and the execution efficiency of multiple tasks to be executed can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer model technology, and in particular to a processing system and method based on cross-network graphic data structure fusion. Background Art

[0002] In the related art, CN118036652A discloses a graph-level representation learning method and a graph classification method for graph structure data, the learning method steps include: constructing a graph-level representation model, the graph-level representation model includes a plurality of absorbing convolution layers connected in sequence, each absorbing convolution layer sequentially performs neighborhood propagation, substructure component extraction, substructure aggregation and topology reconstruction by performing graph convolution operations; obtaining the to-be-processed graph structure data including a node set and an edge set, inputting them into the constructed graph-level representation model, each absorbing convolution layer first uses a graph attention network to calculate the importance value of each node in the node set in the neighborhood information propagation process in the neighborhood propagation stage, and then distinguishes a plurality of high-information nodes with the highest amount of information according to the importance of each node in the substructure component extraction stage, and then uses the high-information nodes to absorb the low-information neighbor nodes to form a substructure in the substructure aggregation stage, and finally obtains a coarsened graph topology by retaining the edges between the substructures centered on the high-information nodes in the topology reconstruction stage, and obtains the final graph-level representation result output after passing through a plurality of absorbing convolution layers. The scheme has the advantages of simple implementation method, low complexity, high learning effectiveness and high efficiency.

[0003] CN118410830A discloses a method for counting subgraphs of a neural network based on graph structure aggregation, which relates to the field of subgraph isomorphism technology. The scheme first traverses the vertices of the query graph and the data graph, records the graph structure formed by each vertex and its neighboring nodes, and then converts the graph structure into a tensor form, participates in the iteration process of the neural network, and finally obtains the number of subgraph matches in the data graph for different query graphs in turn. The scheme not only enriches the feature information of the nodes, but also requires less computing resources and storage space, improves the execution efficiency of the iteration stage of the neural network, and achieves the purpose of reducing the amount of calculation in the aggregation process and reducing counting errors.

[0004] Therefore, the relevant technology can perform operations on graph structures through neural network models, but in the scenario of multi-task parallel execution, the process of each task can constitute a graph data structure, and the operation of each node of each graph data structure requires certain computing resources. The relevant technology is difficult to simplify the graph data structure of the multi-task scenario, making the execution process in this scenario more complicated and difficult to optimize the allocation of computing resources.

[0005] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0006] The present invention provides a processing system and method based on cross-network graphic data structure fusion, which can solve the technical problems that related technologies are difficult to simplify the graphic data structure of multi-task scenarios and difficult to optimize the allocation of computing resources.

[0007] According to a first aspect of the present invention, there is provided a processing system based on cross-network graph data structure fusion, comprising:

[0008] A first acquisition module is used to acquire a set of pending tasks to be executed by a set of computing units, wherein the set of computing units includes a plurality of computing units, each computing unit can call computing resources, the set of pending tasks includes a plurality of pending tasks, each of the pending tasks includes a plurality of steps, and the computing resources include at least one of CPU resources, GPU resources, memory resources, and network bandwidth resources;

[0009] A graph data structure module, used to determine a graph data structure of the task to be executed according to input information, output information and operation information of multiple steps of the task to be executed, wherein the graph data structure includes nodes and connection relationships between nodes, the nodes correspond to the steps of the task to be executed one by one, and the connection relationships are used to represent the data flow between the steps of the task to be executed;

[0010] A fusion node module, used to determine a fusion node in a graph data structure of multiple tasks to be executed according to the input information, the output information and the operation information;

[0011] A target graph data structure module, used for determining a target graph data structure to be fused among multiple graph data structures according to the number of fused nodes;

[0012] A fusion graph data structure module is used to fuse the target graph data structure according to the fusion node of the target graph data structure to be fused, so as to obtain a fusion graph data structure;

[0013] A second acquisition module is used to obtain the computing resource requirement of each node in the fusion graph data structure and predict the computing time;

[0014] An allocation module, configured to allocate a computing unit to each node according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes;

[0015] The execution module is used to process the fused graphic data structure according to the allocated computing units to obtain the execution results of the tasks to be executed corresponding to the multiple target graphic data structures of the fused graphic data structure.

[0016] According to a second aspect of the present invention, a processing method based on cross-network graph data structure fusion is provided, comprising:

[0017] Obtaining a set of pending tasks to be executed by a set of computing units, wherein the set of computing units includes a plurality of computing units, each computing unit can call computing resources, the set of pending tasks includes a plurality of pending tasks, each of the pending tasks includes a plurality of steps, and the computing resources include at least one of CPU resources, GPU resources, memory resources, and network bandwidth resources;

[0018] Determine a graph data structure of the task to be executed according to input information, output information, and operation information of multiple steps of the task to be executed, wherein the graph data structure includes nodes and connection relationships between nodes, the nodes correspond to the steps of the task to be executed one by one, and the connection relationships are used to represent the data flow between the steps of the task to be executed;

[0019] Determine, according to the input information, the output information and the operation information, a fusion node in a graph data structure of multiple tasks to be executed;

[0020] According to the number of the fused nodes, determining a target graph data structure to be fused among the multiple graph data structures;

[0021] According to the fusion node of the target graphic data structure to be fused, the target graphic data structure is fused to obtain a fused graphic data structure;

[0022] Obtaining the computing resource requirement of each node in the fusion graph data structure and predicting the computing time;

[0023] Allocate a computing unit to each node according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes;

[0024] The fused graphic data structure is processed according to the allocated computing units to obtain execution results of tasks to be executed corresponding to a plurality of target graphic data structures of the fused graphic data structure.

[0025] Technical effect: According to the present invention, it is possible to determine nodes with similar characteristics for fusion according to the various characteristics of the nodes of the graph data structure of multiple tasks to be executed, thereby simplifying multiple graph data structures, and facilitating the allocation of computing units for nodes with similar characteristics, and providing corresponding computing resources, thereby optimizing the allocation of computing resources and improving the execution efficiency of multiple tasks to be executed. When determining the fusion score, the similarity between two nodes can be determined from three aspects: input information, output information, and computing information, and the comprehensiveness and objectivity of the fusion score can be improved, so that the similarity between the two nodes and the feasibility of fusion can be accurately described, and when calculating the similarity in the three aspects, the maximum value function and the minimum value function are used to appropriately increase the value of the overall similarity, and to increase the number of fusion nodes to a certain extent, so that the graph data structure is simplified as much as possible, thereby improving the processing efficiency of multiple tasks to be executed. When determining the discrimination score, not only the first proportion of the fusion node in all nodes in the graph data structure is considered, but also the second proportion of the fusion node corresponding to the fusion node in the graph data structure in all nodes of other graph data structures is considered, so as to judge whether the graph data structure is suitable for fusion with other graph data structures, and improve the judgment accuracy. When allocating computing units, whether the node is a fusion node can be considered when setting the objective function, so as to use the total amount of computing resources consumed by the fusion node when executing the task to be executed and the judgment parameters to determine the computing time of any task to be executed at the fusion node when executing multiple tasks at the same time, thereby improving the accuracy of the computing time and providing accurate constraints. When setting the objective function, the total computing time is optimized, the computing units are made more centralized, the data transmission time is reduced, and it is convenient for other tasks to call the computing units.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of a processing system based on cross-network graph data structure fusion according to an embodiment of the present invention is exemplarily shown;

[0029] Figure 2A flowchart of a processing method based on cross-network graph data structure fusion according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0032] Figure 1 A schematic diagram of a processing system based on cross-network graph data structure fusion according to an embodiment of the present invention is exemplarily shown, the system comprising:

[0033] A first acquisition module is used to acquire a set of pending tasks to be executed by a set of computing units, wherein the set of computing units includes a plurality of computing units, each computing unit can call computing resources, the set of pending tasks includes a plurality of pending tasks, each of the pending tasks includes a plurality of steps, and the computing resources include at least one of CPU resources, GPU resources, memory resources, and network bandwidth resources;

[0034] A graph data structure module, used to determine a graph data structure of the task to be executed according to input information, output information and operation information of multiple steps of the task to be executed, wherein the graph data structure includes nodes and connection relationships between nodes, the nodes correspond to the steps of the task to be executed one by one, and the connection relationships are used to represent the data flow between the steps of the task to be executed;

[0035] A fusion node module, used to determine a fusion node in a graph data structure of multiple tasks to be executed according to the input information, the output information and the operation information;

[0036] A target graph data structure module, used for determining a target graph data structure to be fused among multiple graph data structures according to the number of fused nodes;

[0037] A fusion graph data structure module is used to fuse the target graph data structure according to the fusion node of the target graph data structure to be fused, so as to obtain a fusion graph data structure;

[0038] A second acquisition module is used to obtain the computing resource requirement of each node in the fusion graph data structure and predict the computing time;

[0039] An allocation module, configured to allocate a computing unit to each node according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes;

[0040] The execution module is used to process the fused graphic data structure according to the allocated computing units to obtain the execution results of the tasks to be executed corresponding to the multiple target graphic data structures of the fused graphic data structure.

[0041] According to the processing system based on cross-network graph data structure fusion according to the embodiment of the present invention, it is possible to determine nodes with similar characteristics for fusion based on the various characteristics of the nodes of the graph data structure of multiple tasks to be executed, thereby simplifying the multiple graph data structures, and facilitating the allocation of computing units to nodes with similar characteristics, and providing corresponding computing resources, thereby optimizing the allocation of computing resources and improving the execution efficiency of multiple tasks to be executed.

[0042] According to one embodiment of the present invention, the first acquisition module can acquire a set of pending tasks to be executed. The set of pending tasks may include multiple pending tasks. These tasks can be executed in parallel to shorten the execution time and improve the execution efficiency. However, the computing resources of the computing unit set need to be reasonably allocated to make the execution process more efficient.

[0043] According to one embodiment of the present invention, in the graph data structure module, the graph data structure of each task to be executed can be determined, the graph data structure includes nodes, and the connection relationship between the nodes. The nodes correspond to the steps of the task to be executed one by one. Based on the connection relationship, a node can receive the data stream of the previous node and perform operations. After obtaining the operation result, the operation result can be transmitted to the next node in the form of a data stream based on the connection relationship, and the next node continues to perform operations. The connection relationship may include a transmission direction, for example, the transmission direction may include unidirectional transmission and bidirectional transmission. For example, in unidirectional transmission, a node can transmit data to another node in the form of a data stream, while the other node cannot transmit data to the node. In bidirectional transmission, a node can transmit data to another node in the form of a data stream, and the other node can also transmit data to the node in the form of a data stream. Therefore, the graph data structure can be determined according to the steps and processes of the task to be executed, that is, the connection relationship pointing to the node is determined based on the input information, the connection relationship of the node output is determined based on the output information, and the operation mode of the node itself is determined based on the operation information. For example, the node performs addition, multiplication, convolution, etc.

[0044] According to one embodiment of the present invention, each of the above tasks to be executed can determine a graph data structure, and each graph data structure can have relatively similar nodes. Two or more graph data structures can be merged based on these relatively similar nodes, thereby simplifying multiple graph data structures, and facilitating the allocation of computing units to each node, and facilitating the computing units to call processing resources. For example, two similar nodes are merged into the same node, and the two merged nodes have the same operation mode, so the types of computing resources required are the same. Therefore, merging similar nodes into the same node can facilitate the allocation of computing units and the calling of computing resources. For example, if two nodes both need to perform multiplication operations and need to use CPU resources during the operation process, a computing unit that can call CPU resources is allocated to the merged node.

[0045] According to one embodiment of the present invention, based on the input information, output information and operation information, a fusion node in a graph data structure of multiple tasks to be executed is determined, including: determining the input data type, input data amount and input data range of the input information; determining the output data type, output data amount and output data range of the output information; determining the operation type, the number of operation parameters and the operation parameters of the node based on the operation information; determining the fusion node in the graph data structure of multiple tasks to be executed based on the input data type, the input data amount, the input data range, the output data type, the output data amount, the output data range, the operation type, the number of operation parameters and the operation parameters.

[0046] According to one embodiment of the present invention, information such as the input data type, input data amount, and input data range of the input information can be used to determine the situation of the data involved in the operation in the node. If the situation of the data involved in the operation in two nodes is similar, the two nodes may have similarity. The output data type, output data amount, and output data range of the output information can be used to determine the operation result of the node. If the types of the operation results of the two nodes are similar, the two nodes may have similarity. The operation type of the node, the number of operation parameters, and the operation parameters can reflect the operation process of the node. If the operation processes of the two nodes are similar, the two nodes may have similarity.

[0047] According to an embodiment of the present invention, determining the fusion node in the graph data structure of multiple tasks to be executed according to the input data type, the input data amount, the input data range, the output data type, the output data amount, the output data range, the operation type, the number of operation parameters and the operation parameters includes: determining the fusion score between the jth node in the graph data structure of the i-th task to be executed and the tth node in the graph data structure of the s-th task to be executed according to formula (1): :

[0048] (1)

[0049] in, is the feature vector of the input data type of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the input data volume of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the input data range of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the input data type of the tth node in the graph data structure of the sth task to be executed, is the feature vector of the input data volume of the tth node in the graph data structure of the sth task to be executed, is the characteristic vector of the input data range of the tth node in the graph data structure of the sth task to be executed, max is the maximum value function, is the characteristic vector of the output data type of the jth node in the graph data structure of the i-th task to be executed, is the characteristic vector of the output data volume of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the output data range of the jth node in the graph data structure of the i-th task to be executed, is the characteristic vector of the output data type of the tth node in the graph data structure of the sth task to be executed, is the characteristic vector of the output data volume of the tth node in the graph data structure of the sth task to be executed, is the feature vector of the output data range of the tth node in the graph data structure of the sth task to be executed, is the operation type of the jth node in the graph data structure of the i-th task to be executed, is the number of operation parameters of the jth node in the graph data structure of the i-th task to be executed, is the operation type of the tth node in the graph data structure of the sth task to be executed, is the number of operation parameters of the tth node in the graph data structure of the sth task to be executed, , and is a preset weight, and i, j, s, and t are all positive integers; when the fusion score is greater than or equal to the preset score threshold, the j-th node in the graph data structure of the i-th task to be executed and the t-th node in the graph data structure of the s-th task to be executed are determined as fusion nodes.

[0050] According to one embodiment of the present invention, the feature vector of the input data type of each node in the graph data structure of each task to be executed can be determined. For example, the input data type may include an integer type, a floating point type, a Boolean type, etc. Each type may have a corresponding feature vector, and the feature vector corresponding to each type is a preset vector. For example, the feature vector of the integer type is , the floating point type feature vector is , the characteristic vector of Boolean type is .

[0051] According to one embodiment of the present invention, the characteristic vector of the input data amount of each node in the graph data structure of each task to be executed can be determined. For example, the characteristic vector of the input data amount is a 1×1 vector, and the modulus of the vector is the input data amount. For example, if the input data amount is 15 data, the characteristic vector of the input data amount is .

[0052] According to one embodiment of the present invention, the characteristic vector of the input data range of each node in the graph data structure of each task to be executed can be determined. The characteristic vector of the input data range may include multiple components, and the numerical value of each component represents the endpoint value of the interval corresponding to the input data range, and the open and closed state of the interval. For example, the first component represents the left endpoint value of the interval, the second component represents the right endpoint value of the interval, the third component represents the open and closed state of the left endpoint of the interval (for example, 0 represents that the left endpoint is an open interval, and 1 represents that the left endpoint is a closed interval), and the fourth component represents the open and closed state of the right endpoint of the interval (for example, 0 represents that the right endpoint is an open interval, and 1 represents that the right endpoint is a closed interval). For example, if the interval corresponding to the input data range is [0,10], then its corresponding characteristic vector is .

[0053] According to an embodiment of the present invention, the above three feature vectors can form the input information vector of the node. For example, in formula (1), is the input information vector of the jth node in the graph data structure of the i-th task to be executed, is the input information vector of the tth node in the graph data structure of the sth task to be executed, It can represent the minimum gap between two input information vectors, and its denominator can take the maximum value of the modulus of the two input information vectors, which can minimize the relative gap between the two input information vectors. It can also avoid inconsistencies when calculating the fusion score between the j-th node in the graph data structure of the ith task to be executed and the t-th node in the graph data structure of the s-th task to be executed, and when calculating the fusion score between the t-th node in the graph data structure of the s-th task to be executed and the j-th node in the graph data structure of the ith task to be executed (the two should be consistent in theory). It can represent the maximum similarity of the input information vectors of the two nodes, thereby describing the overall similarity of the two nodes in the input information, and appropriately improving the value of the overall similarity, increasing the number of fusion nodes to a certain extent, so as to simplify the graph data structure as much as possible, thereby improving the processing efficiency of multiple tasks to be executed.

[0054] According to one embodiment of the present invention, similarly, the characteristic vector of the output data type of each node in the graph data structure of each task to be executed, the characteristic vector of the output data amount of each node in the graph data structure of each task to be executed, and the characteristic vector of the output data range of each node in the graph data structure of each task to be executed can be determined. Represents the maximum similarity of the output information vectors of two nodes.

[0055] According to one embodiment of the present invention, two parameters, namely the number of operation parameters and the operation type of a node, can constitute the operation information vector of the node, wherein the operation type is the type of operation method, for example, addition operation type, multiplication operation type, convolution operation type, etc., and a specific numerical value can be used to represent the type of operation method. It is the maximum similarity of the two operation information vectors.

[0056] According to an embodiment of the present invention, the minimum value of the difference between the operation parameters of two nodes is calculated. In the example, the operation mode of the node is the weighted sum operation mode, and the operation parameter is the weight used in the weighted sum. When calculating the minimum value of the difference between the operation parameters, the operation parameters can be selected from the operation parameters of the two nodes respectively so that the difference between the operation parameters is Minimize, for example, the operation parameters of the jth node in the graph data structure of the i-th task to be executed are 0.2, 0.3 and 0.5, and the operation parameters of the tth node in the graph data structure of the s-th task to be executed are 0.4, 0.4, 0.2, then you can select a number from each of the two groups of operation parameters to form a team (for example, 0.2 is selected from the first group of operation parameters and 0.4 is selected from the second group of operation parameters), and the operation parameters in each team are substituted into the difference of the above operation parameters for calculation, so as to determine the difference of the operation parameters under this teaming method, and select the teaming method that minimizes the difference of the above operation parameters from multiple teaming methods, for example, , and after multiple comparisons, it is determined that the teaming method of 0.2 and 0.2, 0.3 and 0.4, and 0.5 and 0.4 can minimize the difference in the operation parameters. The difference in the operation parameters calculated by this teaming method is the minimum difference in the operation parameters of the two nodes. If m and n are not equal, that is, the number of operation parameters of the two nodes is inconsistent, the nodes with fewer operation parameters are padded with 0. For example, the operation parameters of the jth node in the graph data structure of the i-th task to be executed are 0.2, 0.3 and 0.5, and the operation parameters of the tth node in the graph data structure of the s-th task to be executed are 0.4 and 0.6. When calculating, the operation parameters of the tth node in the graph data structure of the s-th task to be executed can be set to 0.4, 0.6 and 0. On the other hand, the operation parameters of each node can be formed into a vector, and the modulus of the vector can be solved separately, that is, and , and select the maximum value from them as the denominator, so as to solve the minimum relative difference of the operation parameters of the two nodes, that is, the minimum value of the difference between the operation parameters of the two nodes and and The maximum similarity of the operation parameters of the two nodes is determined by the ratio between the maximum values ​​of , that is, 1 minus the difference between the above ratios, where is the xth operation parameter of the jth node in the graph data structure of the i-th task to be executed, is the yth operation parameter of the tth node in the graph data structure of the sth task to be executed, n is the number of operation parameters of the jth node in the graph data structure of the ith task to be executed, m is the number of operation parameters of the tth node in the graph data structure of the sth task to be executed, x≤n, y≤m, and i, j, s, t, x, y, m and n are all positive integers.

[0057] According to an embodiment of the present invention, the maximum similarity of the two operation information vectors is multiplied by the maximum similarity of the operation parameters of the two nodes to obtain the maximum similarity of the two node operation information.

[0058] According to one embodiment of the present invention, the maximum similarity of the input information vectors of two nodes, the maximum similarity of the output information vectors of two nodes, and the maximum similarity of the operation information of two nodes may be weightedly summed to obtain the fusion score of the two nodes, thereby determining the similarity between the two nodes from three aspects of input information, output information, and operation information, and obtaining a fusion score that can describe the similarity between the two nodes.

[0059] According to an embodiment of the present invention, the fusion scores between the nodes in the graph data structure and the nodes in other graph data structures may be determined in the above manner, thereby determining the fusion nodes in each graph data structure.

[0060] In this way, the similarity between two nodes can be determined from the three aspects of input information, output information and operation information, which improves the comprehensiveness and objectivity of the fusion score, so as to accurately describe the similarity between the two nodes and the feasibility of fusion. When calculating the similarity in the three aspects, the maximum value function and the minimum value function are used to appropriately improve the value of the overall similarity, increase the number of fusion nodes to a certain extent, and simplify the graph data structure as much as possible, thereby improving the processing efficiency of multiple tasks to be executed.

[0061] According to an embodiment of the present invention, in the target graph data structure module, whether the graph data structure is a target graph data structure capable of being fused may be determined based on the number of fused nodes in each graph data structure.

[0062] According to one embodiment of the present invention, based on the number of fused nodes, a target graph data structure to be fused is determined among multiple graph data structures, including: determining a first ratio of fused nodes in the graph data structure of the i-th task to be executed among all nodes; determining a second ratio of fused nodes in the graph data structure of the s-th task to be executed corresponding to the fused nodes in the graph data structure of the i-th task to be executed among all nodes in the graph data structure of the s-th task to be executed; and determining whether the graph data structure of the i-th task to be executed is the target graph data structure to be fused based on the first ratio and the second ratio.

[0063] According to an embodiment of the present invention, determining whether the graphic data structure of the i-th task to be executed is the target graphic data structure to be fused according to the first ratio and the second ratio includes: determining the discrimination score of the graphic data structure of the i-th task to be executed according to formula (2): ,

[0064] (2)

[0065] in, is the first ratio of the fused nodes in the graph data structure of the i-th task to be executed among all the nodes, is the second proportion of the fused nodes in the graph data structure of the sth task to be executed corresponding to the fused nodes in the graph data structure of the ith task to be executed in all the nodes in the graph data structure of the sth task to be executed; if the discriminant score of the graph data structure of the ith task to be executed is greater than or equal to the discriminant score threshold, then the graph data structure of the ith task to be executed is determined to be the target graph data structure to be fused.

[0066] According to one embodiment of the present invention, in formula (2), whether the graph data structure can be used as the target graph data structure can be jointly determined based on the first ratio of fused nodes in the graph data structure and the second ratio of fused nodes in other graph data structures corresponding to the fused nodes in the graph data structure in the other graph data structures. In the case where the first ratio is high, there are more fused nodes in the graph data structure that can be fused with nodes in other graph data structures, and the graph data structure is also easy to merge with other graph data structures into a whole. On the other hand, the second ratio also needs to be considered. If the first ratio of fused nodes in the graph data structure is high, but the fused nodes in other graph data structures corresponding to these fused nodes are too scattered, then the graph data structure is not easy to merge with other graph data structures. For example, a graph data structure includes 10 nodes in total, 9 of which are fused nodes. Therefore, the first ratio is high. However, the 9 fused nodes correspond to fused nodes in 9 other graph data structures respectively, that is, there is only one fused node between the graph data structure and each of the other graph data structures. Then, the graph data structure is not easy to merge with other graph data structures. Therefore, it can be used As another judgment criterion, that is, the maximum value of the second ratio of the fused nodes in the graph data structure of the sth task to be executed corresponding to the fused nodes in the graph data structure of the i-th task to be executed among all the nodes in the graph data structure of the s-th task to be executed, if the maximum value is large, then there is at least one graph data structure in other graph data structures, and there are more fused nodes that can be fused with the graph data structure, that is, there is at least one other graph data structure suitable for fusion with the graph data structure. Therefore, the above first ratio can be multiplied by the maximum value of the second ratio to obtain the discrimination score of the graph data structure of the i-th task to be executed. If the discrimination score is greater than or equal to the discrimination score threshold, it means that both of the above aspects meet the requirements of fusion, and the graph data structure of the i-th task to be executed can be determined as the target graph data structure to be fused.

[0067] In this way, not only the first proportion of the fused nodes in all nodes in the graph data structure is considered, but also the second proportion of the fused nodes corresponding to the fused nodes in the graph data structure in all nodes of other graph data structures is considered, so as to judge whether the graph data structure is suitable for fusion with other graph data structures and improve the judgment accuracy.

[0068] According to one embodiment of the present invention, in the fusion graph data structure module, the fusion nodes in the target graph data structure can be fused, that is, the fusion nodes in different target graph data structures are used as the same node of the fused fusion graph data structure, thereby obtaining a fusion graph data structure.

[0069] According to one embodiment of the present invention, in the second acquisition module, the computing resource demand of each node and the predicted computing time can be obtained. The predicted computing time is the time required to execute the computing process of the node when the computing resource demand of the node is fully met.

[0070] According to one embodiment of the present invention, in the allocation module, a computing unit can be allocated to each node, so that suitable computing resources can be called for processing. Based on the computing resource demand, the predicted computing time, the nodes in the fused graph data structure, and the connection relationship between the nodes, a computing unit is allocated to each node, including: determining the constraint conditions of the computing unit optimization allocation model based on the computing resource demand, the predicted computing time, the nodes in the fused graph data structure, and the connection relationship between the nodes; determining the objective function of the computing unit optimization allocation model based on the predicted computing time and the connection relationship; determining the computing unit that performs computing on each node based on the constraint conditions and the objective function.

[0071] According to one embodiment of the present invention, the operation unit optimization allocation model is a genetic algorithm model or a nonlinear programming model. The present invention does not limit the type of the operation unit optimization allocation model. Constraints and objective functions can be set to find the optimal solution that can maximize the realization of the goal described by the objective function under the constraints of the constraints. The optimal solution is the optimal allocation plan for the operation units.

[0072] According to an embodiment of the present invention, the constraint conditions of the operation unit optimization allocation model are determined according to the operation resource demand, the predicted operation time, the nodes in the fusion graph data structure, and the connection relationship between the nodes, including: determining the constraint conditions of the operation unit optimization allocation model according to formula (3), formula (4), formula (5), formula (6), formula (7), and formula (8),

[0073] (3)

[0074] (4)

[0075] (5)

[0076] (6)

[0077] (7)

[0078] (8)

[0079] in, is the computing resource requirement when the tth node is a non-fused node, The amount of computing resources that can be used by the zth computing unit to operate on the tth node, is the computing resource requirement of the hth task to be executed using the tth node on the tth node when the tth node is a fusion node, max is the maximum value function, is the number of tasks to be executed using the tth node, is the predicted computing time of the hth task to be executed using the tth node on the tth node, The judgment parameter is used to determine whether the g-th task to be executed at the T-th time is executing the operation of the t-th node. is the computing resource requirement of the g-th task to be executed on the t-th node when the t-th node is a fusion node, is the time interval between adjacent moments, is the start time of the operation of the hth task to be executed using the tth node on the tth node, is the time when the operation of the hth task to be executed using the tth node on the tth node ends. is the computation time of the hth task to be executed using the tth node on the tth node, is the computation time of the hth task to be executed using the ath node on the ath node, is the start time of the operation of the gth task to be executed using the tth node on the tth node, is the time when the operation of the g-th task to be executed using the t-th node on the t-th node ends, and if is a conditional function.

[0080] According to one embodiment of the present invention, in formulas (3) and (4), it can be expressed that when the zth operation unit is used to operate the tth node (i.e., the zth operation unit is assigned to the tth node), the zth operation unit can call an operation resource amount greater than or equal to the operation resource requirement of the tth node, so that the operation performed by the tth node can be performed normally. Wherein, in the case where the tth node is a non-fused node, there is only one task to be executed that uses the node, then the operation unit can call an operation resource amount greater than or equal to the operation resource requirement of the node when the task to be executed is executed. If the tth node is a fused node, there are multiple tasks to be executed that need to use the node when executing. Therefore, the operation unit can call an operation resource amount greater than or equal to the maximum value of the operation resource requirements of the node when each task to be executed is executed, so as to meet the operation resource requirement of each task to be executed when the step corresponding to the node is executed.

[0081] According to an embodiment of the present invention, formula (5), formula (6), formula (7), and formula (8) are constraints for describing the operation duration of a node. In formula (5), when the tth node is a fusion node, It can represent the total amount of computing resources consumed by the hth task to be executed using the tth node in the predicted computing time on the tth node. It can be expressed that if the condition is met In the case of , otherwise ,in, represents the computing resource demand of the g-th task to be executed using the t-th node at the T-th time, where The judgment parameter for judging whether the gth task to be executed at the Tth time is using the tth node is as shown in formula (8), that is, if the Tth time is between the start time and the end time of the operation of the gth task to be executed using the tth node on the tth node, then The value of is 1, otherwise it is 0. In other words, if the gth task to be executed is being executed at the Tth moment, then The value of is 1, otherwise it is 0, so That is, the total computing resource demand of all tasks to be executed using the tth node at the Tth moment. If the total computing resource demand is greater than the computing resource that can be called by the zth computing unit, the computing resource supply of each task is allocated according to the demand ratio, that is, is the proportion of the computing resource demand of the hth task to be executed when using the tth node in the total computing resource demand, That is, the computing resource supply that can be provided to the hth task to be executed. If the total computing resource demand is less than or equal to the computing resource that can be called by the zth computing unit, computing resources can be supplied according to the computing resource demand of the hth task to be executed when using the tth node. Then it is the computing resource consumption of the hth task to be executed when using the tth node in the time period between this moment and the next moment. The sum of the computing resource consumption of the hth task to be executed using the tth node in all time periods is equal to the above , that is, the total amount of computing resources consumed by the h-th task to be executed using the t-th node in the predicted computing time on the t-th node.

[0082] According to an embodiment of the present invention, formula (6) can represent the operation time of the hth task to be executed using the tth node on the tth node. If the tth node is not a fused node, the operation time is equal to the predicted operation time. If the tth node is a fused node, the operation time is equal to the total time of all time periods in which the hth task to be executed uses the tth node in formula (5), that is, Formula (7) indicates that the start time of the operation of the hth task to be executed using the tth node on the tth node is equal to the sum of the operation duration of the hth task to be executed on all previous nodes, that is, the operation of the tth node starts immediately after all previous nodes are completed.

[0083] According to an embodiment of the present invention, determining the objective function of the operation unit optimization allocation model according to the predicted operation duration and the connection relationship includes: determining the objective function of the operation unit optimization allocation model according to formula (9) and formula (10),

[0084] (9)

[0085] (10)

[0086] in, is the total computing time of all pending tasks that use the tth node for computing on the tth node, M is the number of nodes in the fusion graph data structure, max is the maximum value function, and minimize is the minimization function.

[0087] According to one embodiment of the present invention, in formula (9), In order to integrate the total computing time of all nodes in the graph data structure, the total computing time can be minimized as the goal, which can improve the overall execution efficiency of multiple tasks to be executed. In formula (10), is the serial number of the zth computing unit, which can minimize the maximum serial number of the allocated computing units, so that the called computing units can be more concentrated and the computing units can be avoided from being too dispersed. This can not only reduce the time for data transmission between different computing units, but also make it easier for other tasks to be executed to call other computing units.

[0088] In this way, when setting the objective function, whether the node is a fusion node can be considered, so as to use the total amount of computing resources consumed by the fusion node when executing the task to be executed and the judgment parameters to determine the computing time of any task to be executed at the fusion node when executing multiple tasks at the same time, thereby improving the accuracy of the computing time and providing accurate constraints. When setting the objective function, the total computing time is optimized, the computing units are more concentrated, the data transmission time is reduced, and it is convenient for other tasks to call the computing units.

[0089] According to one embodiment of the present invention, a solution can be performed based on the above objective function and constraints, so that under the constraints of the constraints, an optimal solution that can maximize the realization of the goal described by the objective function is solved, that is, the optimal allocation plan of the operating units, thereby determining the operating unit that performs operations on each node.

[0090] According to one embodiment of the present invention, in the execution module, the fused graphic data structure can be processed according to the assigned computing units to obtain the execution results of the tasks to be executed corresponding to multiple target graphic data structures of the fused graphic data structure. The tasks to be executed may include data calculation tasks, graphic processing tasks, text processing tasks, etc. The present invention does not impose any restrictions on the types of tasks to be executed.

[0091] According to the processing system based on cross-network graph data structure fusion of the embodiment of the present invention, it is possible to determine nodes with similar characteristics for fusion according to the various characteristics of the nodes of the graph data structure of multiple tasks to be executed, thereby simplifying multiple graph data structures, and facilitating the allocation of computing units for nodes with similar characteristics, and providing corresponding computing resources, thereby optimizing the allocation of computing resources and improving the execution efficiency of multiple tasks to be executed. When determining the fusion score, the similarity between two nodes can be determined from three aspects: input information, output information and computing information, so as to improve the comprehensiveness and objectivity of the fusion score, so as to accurately describe the similarity between the two nodes and the feasibility of fusion, and when calculating the similarity in the three aspects, the maximum value function and the minimum value function are used to appropriately increase the value of the overall similarity, and to increase the number of fusion nodes to a certain extent, so as to simplify the graph data structure as much as possible, thereby improving the processing efficiency of multiple tasks to be executed. When determining the discrimination score, not only the first proportion of the fusion node in all nodes in the graph data structure is considered, but also the second proportion of the fusion node corresponding to the fusion node in the graph data structure in all nodes of other graph data structures is considered, so as to judge whether the graph data structure is suitable for fusion with other graph data structures, and improve the judgment accuracy. When allocating computing units, whether the node is a fusion node can be considered when setting the objective function, so as to use the total amount of computing resources consumed by the fusion node when executing the task to be executed and the judgment parameters to determine the computing time of any task to be executed at the fusion node when executing multiple tasks at the same time, thereby improving the accuracy of the computing time and providing accurate constraints. When setting the objective function, the total computing time is optimized, the computing units are made more centralized, the data transmission time is reduced, and it is convenient for other tasks to call the computing units.

[0092] Figure 2 A flowchart of a processing method based on cross-network graph data structure fusion according to an embodiment of the present invention is exemplarily shown, the method comprising:

[0093] Step S101, obtaining a set of pending tasks to be executed by a set of computing units, wherein the set of computing units includes a plurality of computing units, each computing unit can call computing resources, the set of pending tasks includes a plurality of pending tasks, each of the pending tasks includes a plurality of steps, and the computing resources include at least one of CPU resources, GPU resources, memory resources, and network bandwidth resources;

[0094] Step S102, determining a graph data structure of the task to be executed according to input information, output information, and operation information of multiple steps of the task to be executed, wherein the graph data structure includes nodes and connection relationships between nodes, the nodes correspond to the steps of the task to be executed one by one, and the connection relationships are used to represent the data flow between the steps of the task to be executed;

[0095] Step S103, determining a fusion node in a graph data structure of multiple tasks to be executed according to the input information, the output information and the operation information;

[0096] Step S104, determining a target graph data structure to be fused from among multiple graph data structures according to the number of fused nodes;

[0097] Step S105, fusing the target graph data structure according to the fusion node of the target graph data structure to be fused, to obtain a fused graph data structure;

[0098] Step S106, obtaining the computing resource requirement of each node in the fusion graph data structure and predicting the computing time;

[0099] Step S107, allocating a computing unit to each node according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes;

[0100] Step S108 , processing the fused graphic data structure according to the allocated computing unit to obtain execution results of the tasks to be executed corresponding to the multiple target graphic data structures of the fused graphic data structure.

[0101] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A processing system based on cross-network graph data structure fusion, characterized in that: include: A first acquisition module is used to acquire a set of pending tasks to be executed by a set of computing units, wherein the set of computing units includes a plurality of computing units, each computing unit can call computing resources, the set of pending tasks includes a plurality of pending tasks, each of the pending tasks includes a plurality of steps, and the computing resources include at least one of CPU resources, GPU resources, memory resources, and network bandwidth resources; A graph data structure module, used to determine a graph data structure of the task to be executed according to input information, output information and operation information of multiple steps of the task to be executed, wherein the graph data structure includes nodes and connection relationships between nodes, the nodes correspond to the steps of the task to be executed one by one, and the connection relationships are used to represent the data flow between the steps of the task to be executed; A fusion node module, used to determine a fusion node in a graph data structure of multiple tasks to be executed according to the input information, the output information and the operation information; A target graph data structure module, used for determining a target graph data structure to be fused among multiple graph data structures according to the number of fused nodes; A fusion graph data structure module is used to fuse the target graph data structure according to the fusion node of the target graph data structure to be fused, so as to obtain a fusion graph data structure; A second acquisition module is used to obtain the computing resource requirement of each node in the fusion graph data structure and predict the computing time; An allocation module, configured to allocate a computing unit to each node according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes; The execution module is used to process the fused graphic data structure according to the allocated computing units to obtain the execution results of the tasks to be executed corresponding to the multiple target graphic data structures of the fused graphic data structure.

2. The processing system based on cross-network graph data structure fusion according to claim 1, characterized in that: Determining a fusion node in a graph data structure of multiple tasks to be executed according to the input information, the output information, and the operation information, including: Determine the input data type, input data amount and input data range of the input information; Determine the output data type, output data amount and output data range of the output information; Determine the operation type, the number of operation parameters and the operation parameters of the node according to the operation information; Determine a fusion node in a graph data structure of multiple tasks to be executed based on the input data type, the input data amount, the input data range, the output data type, the output data amount, the output data range, the operation type, the number of operation parameters and the operation parameters.

3. The processing system based on cross-network graph data structure fusion according to claim 2, characterized in that: Determining a fusion node in a graph data structure of multiple tasks to be executed according to the input data type, the input data amount, the input data range, the output data type, the output data amount, the output data range, the operation type, the number of operation parameters, and the operation parameters, includes: According to the formula Determine the fusion score between the jth node in the graph data structure of the i-th task to be executed and the tth node in the graph data structure of the s-th task to be executed ,in, is the feature vector of the input data type of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the input data volume of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the input data range of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the input data type of the tth node in the graph data structure of the sth task to be executed, is the feature vector of the input data volume of the tth node in the graph data structure of the sth task to be executed, is the characteristic vector of the input data range of the tth node in the graph data structure of the sth task to be executed, max is the maximum value function, is the characteristic vector of the output data type of the jth node in the graph data structure of the i-th task to be executed, is the characteristic vector of the output data volume of the jth node in the graph data structure of the i-th task to be executed, is the feature vector of the output data range of the jth node in the graph data structure of the i-th task to be executed, is the characteristic vector of the output data type of the tth node in the graph data structure of the sth task to be executed, is the characteristic vector of the output data volume of the tth node in the graph data structure of the sth task to be executed, is the feature vector of the output data range of the tth node in the graph data structure of the sth task to be executed, is the operation type of the jth node in the graph data structure of the i-th task to be executed, is the number of operation parameters of the jth node in the graph data structure of the i-th task to be executed, is the operation type of the tth node in the graph data structure of the sth task to be executed, is the number of operation parameters of the tth node in the graph data structure of the sth task to be executed, , and is the preset weight, and i, j, s, t are all positive integers; When the fusion score is greater than or equal to the preset score threshold, the jth node in the graph data structure of the ith task to be executed and the tth node in the graph data structure of the sth task to be executed are determined as fusion nodes.

4. The processing system based on cross-network graph data structure fusion according to claim 3, characterized in that: According to the number of the fused nodes, a target graph data structure to be fused is determined from among the multiple graph data structures, including: Determine a first ratio of fused nodes in the graph data structure of the i-th task to be executed among all nodes; Determine a second ratio of the fused nodes in the graph data structure of the sth task to be executed that correspond to the fused nodes in the graph data structure of the ith task to be executed, among all the nodes in the graph data structure of the sth task to be executed; According to the first ratio and the second ratio, it is determined whether the graph data structure of the i-th task to be executed is the target graph data structure to be fused.

5. The processing system based on cross-network graph data structure fusion according to claim 4, characterized in that: Determining whether the graphic data structure of the i-th task to be executed is a target graphic data structure to be fused according to the first ratio and the second ratio includes: According to the formula Determine the discriminant score of the graph data structure for the i-th task to be performed ,in, is the first ratio of the fused nodes in the graph data structure of the i-th task to be executed among all the nodes, is a second ratio of the fused node in the graph data structure of the sth task to be executed corresponding to the fused node in the graph data structure of the ith task to be executed to all nodes in the graph data structure of the sth task to be executed; If the discrimination score of the graph data structure of the i-th task to be executed is greater than or equal to the discrimination score threshold, the graph data structure of the i-th task to be executed is determined as the target graph data structure to be fused.

6. The processing system based on cross-network graph data structure fusion according to claim 1, characterized in that: According to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes, a computing unit is allocated to each node, including: Determining the constraint conditions of the computing unit optimization allocation model according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes; Determining an objective function of an operation unit optimization allocation model according to the predicted operation duration and the connection relationship; According to the constraint conditions and the objective function, a computing unit for computing each node is determined.

7. The processing system based on cross-network graph data structure fusion according to claim 6, characterized in that: According to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes, the constraint conditions of the computing unit optimization allocation model are determined, including: According to the formula Determine the constraints of the optimization allocation model of the computing units, where: is the computing resource requirement when the tth node is a non-fused node, The amount of computing resources that can be used by the zth computing unit to operate on the tth node, is the computing resource requirement of the hth task to be executed using the tth node on the tth node when the tth node is a fusion node, max is the maximum value function, is the number of tasks to be executed using the tth node, is the predicted computing time of the hth task to be executed using the tth node on the tth node, The judgment parameter is used to determine whether the g-th task to be executed at the T-th time is executing the operation of the t-th node. is the computing resource requirement of the g-th task to be executed on the t-th node when the t-th node is a fusion node, is the time interval between adjacent moments, is the start time of the operation of the hth task to be executed using the tth node on the tth node, is the time when the operation of the hth task to be executed using the tth node on the tth node ends. is the computation time of the hth task to be executed using the tth node on the tth node, is the computation time of the hth task to be executed using the ath node on the ath node, is the start time of the operation of the gth task to be executed using the tth node on the tth node, is the time when the operation of the g-th task to be executed using the t-th node on the t-th node ends, and if is a conditional function.

8. The processing system based on cross-network graph data structure fusion according to claim 7, characterized in that: Determining the objective function of the operation unit optimization allocation model according to the predicted operation duration and the connection relationship includes: According to the formula Determine the objective function of the operation unit optimization allocation model, where: is the total computing time of all pending tasks that use the tth node for computing on the tth node, M is the number of nodes in the fusion graph data structure, max is the maximum value function, minimize is the minimization function, and z is the sequence number of the computing unit.

9. A processing method based on cross-network graph data structure fusion, characterized in that: include: Obtaining a set of pending tasks to be executed by a set of computing units, wherein the set of computing units includes a plurality of computing units, each computing unit can call computing resources, the set of pending tasks includes a plurality of pending tasks, each of the pending tasks includes a plurality of steps, and the computing resources include at least one of CPU resources, GPU resources, memory resources, and network bandwidth resources; Determine a graph data structure of the task to be executed according to input information, output information, and operation information of multiple steps of the task to be executed, wherein the graph data structure includes nodes and connection relationships between nodes, the nodes correspond to the steps of the task to be executed one by one, and the connection relationships are used to represent the data flow between the steps of the task to be executed; Determine, according to the input information, the output information and the operation information, a fusion node in a graph data structure of multiple tasks to be executed; According to the number of the fused nodes, determining a target graph data structure to be fused among the multiple graph data structures; According to the fusion node of the target graphic data structure to be fused, the target graphic data structure is fused to obtain a fused graphic data structure; Obtaining the computing resource requirement of each node in the fusion graph data structure and predicting the computing time; Allocate a computing unit to each node according to the computing resource demand, the predicted computing time, the nodes in the fusion graph data structure, and the connection relationship between the nodes; The fused graphic data structure is processed according to the allocated computing units to obtain execution results of tasks to be executed corresponding to multiple target graphic data structures of the fused graphic data structure.

Images