Method, device and equipment for cooperative execution of execution processor, medium and product
By dividing the static computation graph into computation subgraphs and configuring a heterogeneous structure, the resource configuration of the execution processors in the server is optimized, which solves the problem of wasted resources of multiple execution processors and improves the processing efficiency of inference tasks.
Patent Information
- Application Number
- CN202411069018.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-26
AI Technical Summary
In the case of multiple execution processors deployed in a server, the existing technology processes the overall inference task based on only a single execution processor, which leads to a waste of resources.
By loading the network structure file of the static computation graph corresponding to the inference task, it is divided into at least two computation subgraphs. Based on the computation architecture identifier and input/output tensors of the computation subgraphs, mutation factors are configured for the computation subgraphs to form a heterogeneous structure. The execution processors are then optimized to achieve reasonable resource utilization of multiple processors.
This approach enables the rational configuration of multiple execution processors within the server during inference tasks, fully leveraging the resource advantages of each processor and improving the processing efficiency of inference tasks.
Smart Images

Figure CN119025271A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a method, apparatus, device, medium and product for executing processor cooperation. Background Art
[0002] As the number of AI inference scenarios increases, the requirements for processor inference speed are also getting higher and higher.
[0003] At present, when executing reasoning tasks based on servers, developers are not familiar with the underlying operators and do not know the tensor copy information of the calculation graph corresponding to the reasoning task. Usually, a single execution processor in the server is used to complete the processing of the calculation graph corresponding to the entire reasoning task. For example, the entire calculation graph is processed based on the central processing unit or the graphics processing unit. However, when such a configuration method deploys multiple execution processors in the server that can be used to execute reasoning tasks, there is a problem of waste of execution processor resources.
[0004] In order to solve the above problems, it is necessary to improve the method of executing processor configuration execution during the reasoning task processing. Summary of the invention
[0005] Embodiments of the present invention provide a method, apparatus, device, medium and product for executing processors in cooperation, so as to solve the problem in the prior art that when a plurality of execution processors that can be used to execute reasoning tasks are deployed in a server, when reasoning processing is performed on the entire reasoning task based on only a single execution processor, there is a waste of resources of the execution processor in the server.
[0006] In a first aspect, an embodiment of the present invention provides a method for executing processors in coordination, the method comprising:
[0007] Loading a network structure file of a static computation graph corresponding to an inference task; wherein the inference task is a task of how each neural network layer in a neural network processes input data to obtain an output result that meets preset conditions, and the static computation graph includes a plurality of operators, and the operators are associated with the neural network layers and are used to limit processing information of the input data;
[0008] According to the network structure file and the operator cutting constraint condition, the static computation graph is divided into at least two computation subgraphs, and a subgraph file corresponding to the computation subgraph is determined; wherein the subgraph file includes a computation architecture identifier and input and output tensors of the computation subgraph;
[0009] According to the computing architecture identifier and input and output tensors of each computing subgraph, a mutation factor is configured for the computing subgraph to obtain a heterogeneous structure corresponding to the reasoning task; wherein the mutation factor is used to adjust the execution processor corresponding to the computing subgraph, and the execution processor includes a central processing unit or a graphics processing unit;
[0010] For at least one training sample in at least one execution cycle, based on the at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure, determining execution information of each computational subgraph; wherein the execution information includes inference time and inference latency, and the inference latency is the time required to copy the output tensor obtained by the current execution processor to another execution processor;
[0011] Based on the execution information of each execution cycle, a target execution processor of each computation subgraph in the heterogeneous structure is determined, so as to process the received target task based on the target execution processor corresponding to each computation subgraph.
[0012] In a second aspect, an embodiment of the present invention further provides a device for executing a processor in cooperation with the processor, the device comprising:
[0013] A file loading module is used to load a network structure file of a static computation graph corresponding to an inference task; wherein the inference task is a task of how each neural network layer in a neural network processes input data to obtain an output result that meets preset conditions, and the static computation graph includes a plurality of operators, and the operators are associated with the neural network layers and are used to limit processing information of the input data;
[0014] A subgraph partitioning module, used to partition the static computation graph into at least two computation subgraphs according to the network structure file and the operator cutting constraint condition, and determine a subgraph file corresponding to the computation subgraph; wherein the subgraph file includes a computation architecture identifier and input and output tensors of the computation subgraph;
[0015] A heterogeneous structure determination module, configured to configure a mutation factor for each computational subgraph according to the computational architecture identifier and input and output tensors of the computational subgraph, so as to obtain a heterogeneous structure corresponding to the reasoning task; wherein the mutation factor is used to adjust an execution processor corresponding to the computational subgraph, the execution processor including a central processing unit or a graphics processing unit;
[0016] An execution information determination module is used to determine, for at least one training sample in at least one execution cycle, execution information of each computational subgraph based on the at least one training sample, the heterogeneous structure, and a mutation factor corresponding to the computational subgraph in the heterogeneous structure; wherein the execution information includes inference time and inference delay, and the inference delay is the time information required to copy the output tensor obtained by the current execution processor to another execution processor;
[0017] The target task processing module is used to determine the target execution processor of each computing subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computing subgraph.
[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method of executing the processor in cooperation with the processor as described in any one of the embodiments of the present invention is implemented.
[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for executing in cooperation with an execution processor as described in any one of the embodiments of the present invention.
[0020] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the method for executing the processor in cooperation as described in any one of the embodiments of the present invention.
[0021] In an embodiment of the present invention, by loading a network structure file of a static computational graph corresponding to an inference task; dividing the static computational graph into at least two computational subgraphs according to the network structure file and the operator cutting constraint, and determining the subgraph file corresponding to the computational subgraph; configuring a mutation factor for the computational subgraph according to the computational architecture identifier and the input and output tensors of each computational subgraph, and obtaining a heterogeneous structure corresponding to the inference task; for at least one training sample in at least one execution cycle, determining the execution information of each computational subgraph based on at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure; determining the target execution processor of each computational subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computational subgraph. In the present technical solution, at least two computational subgraphs can be obtained by loading a static computational graph corresponding to an inference task and performing subgraph cutting on the static computational graph, and on this basis, each computational subgraph can be configured on each execution processor for simulation operation. Among them, in the process of executing each computational subgraph and optimizing the configuration of the processor, the mutation of the computational subgraph on different execution processors can be controlled by setting the computational architecture identifier and mutation factor corresponding to each computational subgraph, so as to obtain the execution information of the computational subgraph on each execution processor. After the configuration of the execution processor corresponding to each computational subgraph is iteratively optimized for many times, the target execution processor corresponding to each computational subgraph is obtained according to the execution information corresponding to each computational subgraph in each iterative optimization process, that is, the execution processor with the shortest processing time for processing the computational subgraph is used as the target execution processor, and then the target heterogeneous structure corresponding to the reasoning task is determined according to the target execution processors corresponding to all the computational subgraphs, so as to realize the optimal resource configuration of at least two execution processors in the server during the execution of the reasoning task, so as to directly process the computational subgraph in the target task based on the determined target heterogeneous structure when processing the target task of the same type as the reasoning task. The present invention solves the problem in the prior art that, in the case where multiple execution processors that can be used to execute reasoning tasks are deployed in a server, when the overall reasoning task is reasoned and processed only based on a single execution processor, there is a waste of resources of the execution processor in the server. By sub-cutting the static calculation graph corresponding to the reasoning task and allocating at least one obtained calculation sub-graph to a reasonable execution processor for processing, the reasoning task is processed jointly based on at least two execution processors in the server. In this way, in the process of executing the reasoning task, multiple execution processors in the server are rationally configured, the resource advantages of each execution processor are fully utilized, and the processing efficiency of the reasoning task is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 A flowchart of a method for executing a processor in cooperation with an embodiment of the present invention;
[0024] Figure 2 A flowchart of a specific example of a method for executing a processor in cooperation with an embodiment of the present invention;
[0025] Figure 3 A flowchart of a method for executing a processor in cooperation with an embodiment of the present invention;
[0026] Figure 4 A flowchart for iteratively optimizing the configuration of an execution processor corresponding to a computation subgraph provided by an embodiment of the present invention;
[0027] Figure 5 A schematic diagram of the structure of a device for executing in cooperation with a processor according to an embodiment of the present invention;
[0028] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.
[0029] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0031] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, transmission, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present invention, certain software, components or models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.
[0032] Before elaborating on the technical solution in detail, a brief introduction to the application scenarios of the technical solution is first given to facilitate a clearer understanding of the technical solution.
[0033] In the prior art, when processing the static calculation graph of the reasoning task, since the developers are not familiar with the underlying operators in the deep learning network architecture and the heterogeneous copy information of the input and output data in the static calculation graph, the entire static calculation graph is usually deployed on the same execution processor to complete the reasoning calculation of the entire static calculation graph, for example, the entire static calculation graph is deployed on the central processing unit (CPU), or the entire static calculation graph is deployed on the graphics processing unit (GPU). The defect of such a setting is that if the GPU processes the entire static calculation graph, there will be a problem of wasting the idle computing resources of the CPU. If the CPU processes the entire static calculation graph, the resources of the GPU will be limited, resulting in the problem of idle GPU resources waiting and CPU operation blocking. In order to achieve the reasonable use of the execution processor in the server when processing the reasoning task, it is necessary to optimize the configuration of at least two execution processors in the service according to the static calculation graph corresponding to the reasoning task, so as to jointly process the reasoning task based on at least two execution processors, solve the problem of resource waste in the execution process of the reasoning task, and improve the processing efficiency of the same type of reasoning tasks.
[0034] In order to facilitate the understanding of the present technical solution, the terms in the present technical solution are briefly introduced here. The calculation graph mentioned in the present technical solution is a graphical model for describing data calculation, which is commonly used in the forward propagation and backward propagation of neural networks. In the calculation graph, the nodes represent mathematical operations, and the edges represent the dependencies between the operation results, forming a directed acyclic graph. Based on the calculation graph, the entire calculation process can be completed through a series of calculations and data transfers between nodes. The calculation graph mainly includes static calculation graphs and dynamic calculation graphs. In the static calculation graph, the structure of the entire calculation graph cannot be modified after creation, that is, or, the calculation graph is static and will not change with the change of data or neural network parameters, which makes the static graph calculation graph can be effectively optimized, such as, the operators in the static calculation graph are fused, constant folded or redundant calculations are deleted, so as to improve the calculation efficiency of the reasoning task based on the static calculation graph. The dynamic calculation graph is to construct the calculation graph according to the specific input data at runtime. Unlike the static calculation graph, the dynamic calculation graph will be rebuilt each time it is executed, and the present technical solution mainly processes the static calculation graph corresponding to the reasoning task.
[0035] Embodiment 1
[0036] Figure 1 Flow chart of the method for executing the processor in cooperation with the embodiment of the present invention. Figure 1 As shown, the method for executing the processor in cooperation includes:
[0037] S101. Load the network structure file of the static computation graph corresponding to the reasoning task.
[0038] Among them, the so-called reasoning task refers to the process of predicting, classifying or generating new content for unseen data based on the previously learned rules and existing knowledge after the training is completed.
[0039] In this technical solution, the inference task is the task of how each neural network layer in the neural network processes the input data to obtain an output result that meets the preset conditions. The static calculation graph includes multiple operators, which are associated with the neural network layer and are used to limit the processing information of the input data. The so-called operator refers to the basic mathematical operation used in the neural network layer. For example, the operator can be a fully connected layer, a convolutional layer, a pooling layer in the neural network, or it can also be a computing unit containing operation logic such as addition, subtraction, multiplication and division. The network structure file refers to a structure file containing the processing logic corresponding to the static calculation graph corresponding to the inference task.
[0040] For example, the reasoning task may be to classify images of different categories so that after training is completed, when a new image is input into the model, the image category corresponding to the new image can be accurately determined.
[0041] Optionally, loading a network structure file of a static computation graph corresponding to the reasoning task includes: generating a static computation graph based on the acquired reasoning task; and loading the network structure file of the static computation graph when determining that heterogeneous verification conditions exist in the static computation graph.
[0042] In this technical solution, in order to achieve the optimal configuration of the execution processors in the server when executing reasoning tasks, after loading the static computation graph corresponding to the reasoning task, the static computation graph is heterogeneously processed by reasonably allocating at least two execution processors, so as to improve the overall processing efficiency of the static computation graph while ensuring that the computing resources of each execution processor are maximized.
[0043] It can be understood that the heterogeneous structure of the CPU and GPU in the present technical solution is only used as an example. In actual applications, the heterogeneous structure in the server may also be a combination of other types of execution processors. For example, the heterogeneous structure in the server may be a combination of any at least two execution processors of a neural processing unit (NPU), a tensor processing unit (TPU), a CPU and a GPU.
[0044] In a specific example, Figure 2 As shown, taking the processing of reasoning tasks based on a trained large model as an example, after obtaining the reasoning task, the deep learning framework in the large model can automatically generate a static calculation graph corresponding to the reasoning task. Furthermore, before heterogeneous processing of the static calculation graph, it is necessary to first detect whether the server meets the heterogeneous conditions, and only when the server meets the heterogeneous verification conditions, the static calculation graph is heterogeneously processed based on at least two execution processors in the server. Specifically, the type of the execution processor in the server is judged to ensure that at least two execution processors required in the static calculation graph processing process are deployed in the server. If only a single execution processor is deployed in the server, such as only a GPU or CPU, the input tensor of the static calculation graph is copied to the GPU or CPU for reasoning task processing.
[0045] If the server is deployed with two execution processors that can be used to perform reasoning tasks, it is necessary to detect the resource occupancy rate or task processing status of each execution processor. If the resource occupancy rate of at least two execution processors required by the static calculation graph is low and / or the task processing status is idle, it is determined that the server meets the heterogeneous processing conditions of the static calculation graph, and on this basis, the network structure file corresponding to the static calculation graph is loaded.
[0046] On the basis of the above, a static computation graph corresponding to the reasoning task is loaded so that at least two execution processors in the server jointly process the static computation graph according to at least one operator in the static computation graph and the processing logic between the input and output data associated with each operator.
[0047] S102. Divide the static computation graph into at least two computation subgraphs according to the network structure file and the operator cutting constraints, and determine the subgraph files corresponding to the computation subgraphs.
[0048] Among them, the operator cutting condition refers to the constraint conditions for fusing multiple operators in the static calculation graph and then cutting the operators obtained after the fusion. The calculation subgraph refers to the subgraph obtained after the static calculation graph is divided into subgraphs, and the calculation subgraph may contain one or more operators, as well as the input and output data corresponding to each operator. Among them, the input and output data of the operators in the present technical solution are all represented in the form of tensors. The subgraph file includes the computing architecture identifier and input and output tensors of the computing subgraph. The computing architecture identifier is used to characterize the execution processor that the computing subgraph needs to use during the processing.
[0049] It should be noted that since the network structure file corresponding to the static computation graph includes a large number of operators, in order to more conveniently perform subgraph cutting on the static computation graph, the operators in the static computation graph can be first fused. Figure 2 , the static computation graph includes 200 operators, and these 200 operators are fused to obtain 150 operators, and the static computation graph composed of these 150 operators is equivalent to the static computation graph composed of the 200 operators before fusion, and then the static computation graph after the operator fusion is sub-graph cut to obtain at least two computation sub-graphs. For example, the operators in the static computation graph can be fused based on at least one operator fusion method, such as merging multiple convolution kernels into a larger convolution kernel to reduce the amount of calculation; fusing fixed collocation operators into a larger operator to simplify the number of operators; merging multiple continuous neural network layers into a larger network layer to reduce memory access and computing overhead; and computing multiple independent neural network operators in parallel to improve the overall computing speed.
[0050] It is understandable that there are many ways to fuse operators in actual applications, and the operators in the static calculation graph can be fused according to actual conditions. In this technical solution, the way of operator fusion is not specifically limited.
[0051] Optionally, the static computation graph is divided into at least two computation sub-graphs according to the network structure file and operator cutting constraints, including: dividing the network structure file according to the execution order corresponding to each operator in the static computation graph, the corresponding input and output tensors, the execution order based on the operator processing of the input data, whether the operator is on a branch node on the static computation graph, and the operator number threshold corresponding to each computation sub-graph, to obtain at least two computation sub-graphs.
[0052] In a specific example, when the static computation graph is cut according to the network structure file and the operator cutting conditions, the cutting can be performed based on the input and output tensors corresponding to each operator according to the execution order of each operator in the static computation graph. For example, for operator 1 and operator 2, the output tensor of operator 1 is the input tensor corresponding to operator 2. If the tensor size of the output tensor of operator 1 is larger than the preset tensor size, operator 1 and operator 2 are not cut, so that operator 1 and operator 2 can be divided into the same computation subgraph, and then the operator 1 and operator 2 are processed on the same execution processor.
[0053] The advantage of this setting is that when the output tensor of operator 1 is large, if operator 1 and operator 2 are cut, operator 1 and operator 2 may be divided into different calculation subgraphs, and then different execution processors are used to execute the calculation subgraphs corresponding to operator 1 and operator 2. In this process, the output tensor of operator 1 needs to be copied from the current execution processor to another execution processor for execution. Therefore, when calculating the execution time corresponding to operator 1, the copy time of the calculation subgraph corresponding to operator 1 needs to be considered. When the tensor size of the output tensor of operator 1 is large, the copy time consumed is also long, which leads to the overall processing time of the static calculation graph of the entire reasoning task. When the tensor size of the output tensor of operator 1 is small, the copy time of the output tensor can be ignored, and it will not affect the processing time of the overall static calculation graph. Therefore, for operators with large output tensors, the non-cutting method is adopted, and the two adjacent operators will be cut only when the output tensor of the operator is small.
[0054] Exemplarily, the cutting basis can also be determined based on whether the operator is at a branch node on a static computational graph. For each operator, corresponding identification information can be set. For example, if operator 1 and operator 2 are serial structures, the identification information carried by operator 1 and operator 2 is a serial label. If operator 3 and operator 4 are parallel structures, operator 3 and operator 4 respectively carry identification information as a parallel label. In actual situations, there is a dependency relationship between operators of a serial structure, but there is no dependency structure between operators of a parallel structure. Therefore, in this technical solution, operators carrying parallel labels are cut.
[0055] In another example, the number of consecutive operators may be used as a basis for cutting. For example, the maximum number of consecutive operators is set to 30. When 30 consecutive operators are detected, the 30th operator and the 31st operator are cut.
[0056] In another example, the operator attributes can also be used as the basis for cutting. For example, for a convolution type operator, it is obviously suitable for GPU processing. Based on this, it can be marked in the identification information of the convolution operator to cut the operator carrying the convolution operator identification, so that the convolution operator can be allocated to GPU processing in the subsequent execution processor allocation process.
[0057] It should be noted that in this technical solution, there is no specific limitation on the cutting method of the operator, and it can be set according to actual needs.
[0058] Optionally, determining the subgraph file corresponding to the computational subgraph includes: determining file content in the subgraph file corresponding to the computational subgraph based on a computational architecture identifier and input and output tensors of at least one operator in each computational subgraph.
[0059] For details, see Figure 2 After the static computational graph is cut and processed to obtain at least one computational subgraph, any computational subgraph is taken as an example to generate a Json file (i.e., a subgraph file) corresponding to the computational subgraph, and the subgraph information corresponding to the computational subgraph is marked in the subgraph file. For example, at least one operator in the computational subgraph, the input and output tensors of each operator, and the tensor size of the input and output tensors are determined, and the input and output tensors corresponding to the computational subgraph are determined based on the input and output tensors of all operators in the computational subgraph, so that when the computational subgraph is used later, the computational architecture identifier and mutation factor, etc. are configured for the computational subgraph based on the file content of the subgraph file of the computational subgraph.
[0060] S103. According to the computing architecture identifier and input and output tensors of each computing subgraph, a mutation factor is configured for the computing subgraph to obtain a heterogeneous structure corresponding to the reasoning task.
[0061] The mutation factor is used to adjust the execution processor corresponding to the computation subgraph, and the execution processor includes a central processing unit or a graphics processing unit. The so-called heterogeneous structure refers to a structure in which at least two computation subgraphs are allocated to at least two execution processors in a server.
[0062] In practical applications, different execution processors have different processing capabilities for different computational subgraphs. In order to achieve the purpose of optimally configuring the reasoning task based on the execution processor in the server, before finally determining to perform computational processing on each computational subgraph, the processing efficiency of different computational subgraphs can be determined based on each execution processor. Based on this, after the static computational graph is subdivided into at least two computational subgraphs, a corresponding computational architecture identifier can be configured in advance for each computational subgraph to characterize the execution processor used when executing each computational subgraph. At the same time, based on the operators contained in each computational subgraph and the input and output tensors of the operators, the processing time of the execution processor when processing the computational subgraph can be estimated.
[0063] On this basis, for each computational subgraph, in order to determine the optimal execution processor corresponding to the current computational subgraph, it is necessary to calculate the processing time of the current computational subgraph on different execution processors respectively, so as to determine the target execution processor corresponding to the current computational subgraph from at least two execution processors according to the processing time. However, since it is impossible to accurately evaluate the target execution processor suitable for deployment of each computational subgraph before actually running each computational subgraph, the corresponding mutation factor can be configured for each computational subgraph to adjust the execution processor corresponding to the current computational subgraph until the target execution processor corresponding to the current computational subgraph is determined, so as to obtain a heterogeneous structure corresponding to the reasoning task according to the target execution processors corresponding to all computational subgraphs.
[0064] In this process, mutation factors are configured for the computational subgraph according to the computational architecture identifier and input-output tensors of each computational subgraph to obtain a heterogeneous structure corresponding to the reasoning task, including: when the computational architecture identifier of the computational subgraph is the first identifier, a first mutation factor is configured for the computational subgraph; when the computational architecture identifier of the computational subgraph is the second identifier, a second mutation factor is configured for the computational subgraph; based on the mutation factors corresponding to each computational subgraph, the input-output tensors, and the processing order corresponding to each computational subgraph, the heterogeneous structure corresponding to the reasoning task is determined.
[0065] The computing architecture identifier corresponding to the computing subgraph is set according to whether the computing subgraph can be processed on any execution processor of the server.
[0066] Specifically, if any execution processor can process the computation subgraph, the computation architecture identifier corresponding to the computation subgraph is set to the first identifier; if the computation subgraph is only applicable to a certain execution processor, the computation architecture identifier of the computation subgraph is set to the second identifier. Furthermore, a corresponding mutation factor can be set for the computation subgraph according to the computation architecture identifier corresponding to the computation subgraph.
[0067] Exemplarily, when the computing architecture identifier of the computing subgraph is the first identifier, the first mutation factor corresponding to the computing subgraph can be set, for example, the first mutation factor is 0.5, indicating that in the process of determining the target execution processor corresponding to the computing subgraph, the probability of the computing subgraph being mutated and deployed from the first execution processor to the second execution processor for execution, or from the second execution processor to the first execution processor for execution is equal. When the computing architecture identifier of the computing subgraph is the second identifier, the second mutation factor corresponding to the computing subgraph is set accordingly, for example, the second mutation factor is set to 0, which is used to limit the computing subgraph to be executed only on the currently deployed execution processor in the process of determining the target execution processor corresponding to the computing subgraph, and will not mutate to other execution processors.
[0068] It should be noted that the first identifier and the second identifier can be used to set the computing architecture identifier of the computing subgraph according to the processing results of the historical execution processor on different operators.
[0069] Exemplarily, taking the server including the first execution processor and the second execution processor as an example, for a computational subgraph that is not obviously suitable for a certain execution processor, a first identifier is set for the computational subgraph, and a first mutation factor is set to 0.5. In the process of determining the target execution processor corresponding to the computational subgraph, the computational subgraph is continuously mutated between the first execution processor and the second execution processor, and the processing time of the computational subgraph by the first execution processor and the second execution processor during each mutation is recorded, so that after the mutation cycle ends, the execution processor with the shortest processing time is used as the target execution processor of the computational subgraph.
[0070] For example, for a computation including a convolution operator which is obviously suitable for GPU processing, Figure 1 For example, you can set the Figure 1 The corresponding second identifier, and set the second mutation factor to 0, and initially calculate the sub Figure 1 When deploying, directly add the compute Figure 1 Deployed on GPU for processing, and due to the limitation of the second mutation factor, the computation Figure 1 During the optimization process of the target execution processor, the calculation sub Figure 1 It will always be deployed on the GPU and will not be mutated to other execution processors.
[0071] Or, by calculating Figure 2 For example, the calculation Figure 2 With calculator Figure 3 are connected, and the calculation sub Figure 2 The output tensor of is the operator Figure 3 The input tensor of . Figure 2contains at least one operator. The size of the operator's input and output tensors can be inferred based on the size of each operator's input and output tensors. Figure 2 The corresponding input and output tensor sizes are in the operator Figure 2 When the output tensor size of is larger than the preset tensor size, if the operator Figure 2 When the output tensor of is moved from the first execution processor to the second execution processor for processing, that is, based on the second execution processor's Figure 2 When processing, the calculation sub Figure 2 The output tensor of is copied from the first execution processor to the second execution processor. During the copying process, due to the computation Figure 2 The output tensor of is large in size, which will take a certain amount of time to copy, resulting in an increase in the total processing time of the entire static computation graph in the server-based heterogeneous architecture. Figure 2 When setting the compute architecture identifier, you can also Figure 2 The computing architecture identifier of the computing sub-module is set to the second identifier, and the second mutation factor is set to 0 to ensure that the computing sub-module Figure 2 and calculator Figure 3 Deployed on the same execution processor for processing, it improves the overall processing efficiency of the entire computational static graph by reducing the copying time consumed when the computational subgraph is copied between different execution processors.
[0072] Of course, the above examples set the value of the mutation factor for the computational subgraph for illustrative purposes. In practical applications, the mutation factor corresponding to each computational subgraph can be set according to actual needs. Among them, the larger the mutation factor, the greater the mutation probability of the computational subgraph between different execution processors; the smaller the mutation factor, the smaller the mutation probability of the computational subgraph between different execution processors. At the same time, when setting the first identifier or the second identifier corresponding to the computational subgraph, it can be set accordingly according to actual needs, or it can be set according to the historical processing experience of the execution processor. In this technical solution, there is no specific limitation on the setting of the computational architecture identifier and mutation factor of each computational subgraph.
[0073] S104: For at least one training sample in at least one execution cycle, based on the at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure, determine the execution information of each computational subgraph.
[0074] The execution information includes the inference time and the inference delay. The inference delay is the time required to copy the output tensor obtained by the current execution processor to another execution processor. The inference time corresponding to the computational subgraph is the processing time of the computational subgraph by the currently deployed execution processor.
[0075] In practical applications, the target processor corresponding to each computational subgraph can be determined by iteratively training at least two computational subgraphs associated with the inference task. In this process, each complete iterative training process is regarded as an execution cycle, at least one iterative optimization is implemented in each execution cycle, and all computational subgraphs included in each iterative optimization process are regarded as a training sample. Furthermore, based on at least one training sample, a heterogeneous structure, and a mutation factor corresponding to the computational subgraph in the heterogeneous structure, the execution information of each computational subgraph is determined.
[0076] For example, see Figure 2 , taking the subgraph cutting of the static computation graph corresponding to the reasoning task to obtain 10 computation subgraphs as an example, the execution cycle corresponding to the reasoning task is set to 5 execution cycles, each execution cycle includes 100 iterative optimization processes (the number of optimization units is configured), and each execution of the entire static computation graph is regarded as a complete iteration process.
[0077] On this basis, before processing the entire reasoning task, the heterogeneous structure corresponding to the static calculation graph of the reasoning task is determined, such as setting a corresponding calculation architecture identifier and mutation factor for each calculation subgraph, and randomly assigning each calculation subgraph to at least two execution processors in the server for processing, so as to obtain a heterogeneous structure corresponding to the static calculation graph. Further, in each iteration process, according to the initial setting of the calculation architecture identifier and mutation factor carried by each calculation subgraph, the processing of each calculation subgraph is completed on the corresponding execution processor, and the execution information corresponding to each calculation subgraph is recorded. Further, according to the processing result of the first round of iteration, the calculation architecture identifier and mutation factor of each calculation subgraph are updated, so that in the next round of iteration, the execution processor corresponding to each calculation subgraph is re-determined based on the updated calculation architecture identifier and mutation factor, until 100 iterations of optimization are completed, and the execution information corresponding to each calculation subgraph in the 100 iterations of optimization is recorded (that is, in the iterative optimization process, the allocation method is optimized through the continuous mutation of the genetic algorithm, until the fixed reasoning method enters the normal reasoning when the optimal state of convergence is reached).
[0078] Based on similar operations, after completing an execution cycle of the reasoning task, the iterative optimization of the next execution cycle is continued until 5 execution cycles are completed, and the target executor corresponding to each computational subgraph is determined according to the execution information corresponding to each computational subgraph recorded in each execution cycle, so as to obtain the target heterogeneous structure corresponding to the reasoning task according to the target executor of each computational subgraph. Among them, the target heterogeneous structure is used to characterize the target execution processor applicable to each computational subgraph associated with the reasoning task.
[0079] S105. Determine a target execution processor for each computation subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computation subgraph.
[0080] The target task refers to a reasoning task of the same task type as the reasoning task in the training process. For example, if the reasoning task in the training process is an image classification task, then the target task is a reasoning task with image classification as the goal.
[0081] Specifically, for reasoning tasks, the target execution processor corresponding to each computational subgraph can be determined based on the execution information corresponding to each computational subgraph recorded in each execution cycle, so that the corresponding computational subgraphs can be executed separately based on the target execution processor corresponding to each computational subgraph until the entire reasoning task is completed.
[0082] Optionally, after obtaining the target execution processor corresponding to each computational subgraph, the method further includes: receiving a target task; processing the target task based on the target execution processor corresponding to each computational subgraph, and outputting a target result of the target task.
[0083] Among them, the target result refers to the reasoning result obtained by reasoning and analyzing the task information in the target task.
[0084] On this basis, when a target task of the same type as the reasoning task is received, if the computational subgraph contained in the target task is similar or consistent with the computational subgraph associated with the reasoning task, the target execution processor corresponding to each computational subgraph in the reasoning task is directly used as the target processor corresponding to the corresponding computational subgraph in the target task, and the target heterogeneous structure of the target task in the server is obtained, and then the target task is quickly processed to obtain the target result corresponding to the target task.
[0085] The advantage of this setting is that the target execution processor used by each computational subgraph is the most matching execution processor, which can give full play to the resource advantages of different execution processors in the server, solve the problem of resource waste in the process of a single execution processor performing a complete reasoning task, and at the same time improve the overall processing efficiency of the entire reasoning task.
[0086] In an embodiment of the present invention, by loading a network structure file of a static computational graph corresponding to an inference task; dividing the static computational graph into at least two computational subgraphs according to the network structure file and the operator cutting constraint, and determining the subgraph file corresponding to the computational subgraph; configuring a mutation factor for the computational subgraph according to the computational architecture identifier and the input and output tensors of each computational subgraph, and obtaining a heterogeneous structure corresponding to the inference task; for at least one training sample in at least one execution cycle, determining the execution information of each computational subgraph based on at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure; determining the target execution processor of each computational subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computational subgraph. In the present technical solution, at least two computational subgraphs can be obtained by loading a static computational graph corresponding to an inference task and performing subgraph cutting on the static computational graph, and on this basis, each computational subgraph can be configured on each execution processor for simulation operation. Among them, in the process of executing each computational subgraph and optimizing the configuration of the processor, the mutation of the computational subgraph on different execution processors can be controlled by setting the computational architecture identifier and mutation factor corresponding to each computational subgraph, so as to obtain the execution information of the computational subgraph on each execution processor. After the configuration of the execution processor corresponding to each computational subgraph is iteratively optimized for many times, the target execution processor corresponding to each computational subgraph is obtained according to the execution information corresponding to each computational subgraph in each iterative optimization process, that is, the execution processor with the shortest processing time for processing the computational subgraph is used as the target execution processor, and then the target heterogeneous structure corresponding to the reasoning task is determined according to the target execution processors corresponding to all the computational subgraphs, so as to realize the optimal resource configuration of at least two execution processors in the server during the execution of the reasoning task, so as to directly process the computational subgraph in the target task based on the determined target heterogeneous structure when processing the target task of the same type as the reasoning task. The present invention solves the problem in the prior art that, in the case where multiple execution processors that can be used to execute reasoning tasks are deployed in a server, when the overall reasoning task is reasoned and processed only based on a single execution processor, there is a waste of resources of the execution processor in the server. By sub-cutting the static calculation graph corresponding to the reasoning task and allocating at least one obtained calculation sub-graph to a reasonable execution processor for processing, the reasoning task is processed jointly based on at least two execution processors in the server. In this way, in the process of executing the reasoning task, multiple execution processors in the server are rationally configured, the resource advantages of each execution processor are fully utilized, and the processing efficiency of the reasoning task is improved.
[0087] Based on the above embodiments, optionally, the execution information of each computational subgraph is determined to be refined based on at least one training sample, a heterogeneous structure, and a mutation factor corresponding to the computational subgraph in the heterogeneous structure. Accordingly, the present application proposes the following embodiments:
[0088] Figure 3 Flow chart of the method for executing the processor in cooperation with the embodiment of the present invention. Figure 3 As shown, the method for executing the processor in cooperation includes the following steps:
[0089] S201. Load the network structure file of the static computation graph corresponding to the reasoning task.
[0090] S202. Divide the static computation graph into at least two computation subgraphs according to the network structure file and the operator cutting constraints, and determine the subgraph files corresponding to the computation subgraphs.
[0091] S203. According to the computing architecture identifier and input and output tensors of each computing subgraph, a mutation factor is configured for the computing subgraph to obtain a heterogeneous structure corresponding to the reasoning task.
[0092] S204: For at least one training sample in at least one execution cycle, process the current training sample based on at least one computation subgraph in the heterogeneous structure to obtain first execution information of each computation subgraph.
[0093] Among them, the first execution information can be understood as the corresponding inference duration and inference latency when the computation subgraph is processed on the currently deployed execution processor.
[0094] In the present technical solution, at least one execution cycle may be set for determining the target execution processor corresponding to each computational subgraph, and the number of iterations for optimizing the configuration of the execution processor of each computational subgraph in each execution cycle may be set.
[0095] Exemplarily, taking the sub-graph cutting of the static computation graph corresponding to the reasoning task to obtain 10 computation sub-graphs as an example, the execution cycle corresponding to the reasoning task is set to 5 execution cycles, each execution cycle includes 100 iterative optimization processes, and each execution of the entire static computation graph is regarded as a complete iterative process. Accordingly, all the computation sub-graphs contained in the static computation graph are regarded as a training sample.
[0096] Taking one of the training samples in one of the execution cycles as an example, before processing the entire inference task, determine the heterogeneous structure corresponding to the static computation graph of the inference task, such as setting the corresponding computation architecture identifier and mutation factor for each computation subgraph in the current training sample, and randomly assigning each computation subgraph to at least two execution processors in the server for processing, to obtain the heterogeneous structure corresponding to the inference task of the current training sample, see Figure 4 .
[0097] Furthermore, according to the initial settings of the computing architecture identifier and mutation factor carried by each computing subgraph in the current training sample, the processing of each computing subgraph is completed on the corresponding execution processor, and the first execution information corresponding to each computing subgraph is recorded, that is, the inference time and inference delay of each computing subgraph in the current training sample on the corresponding execution processor are determined.
[0098] It should be noted that, in the process of executing mutations, the CPU / GPU may be unable to process the operators in the assigned computational subgraph. In this case, the computational subgraph can be rolled back to the Json file configuration before the mutation cycle. For example, for computational subgraph A, when computational subgraph A includes operators that the GPU cannot process, and computational subgraph A is suitable for CPU processing, when computational subgraph A mutates from the first execution processor to the second execution processor, the GPU cannot process computational subgraph A normally, and the configuration of the execution processor corresponding to computational subgraph A is rolled back to the Json configuration before the mutation cycle, that is, computational subgraph A continues to be processed based on the CPU in the next iteration, and the mutation factor corresponding to computational subgraph A is changed to 0, so that computational subgraph A no longer mutates, and computational subgraph A is always processed by the CPU.
[0099] The advantage of this setting is that computational subgraphs that contain operators that cannot be effectively processed by the execution processor will no longer be mutated, which reduces the number of subgraph mutations, improves the ability to determine the target execution processor corresponding to each computational subgraph, and uses the execution processor that best matches the computational subgraph as the target processor.
[0100] In this process, the current training sample is processed based on at least one computing subgraph in the heterogeneous structure to obtain the first execution information of each computing subgraph, including: based on the execution processor deployed by each computing subgraph in the heterogeneous structure, the current training sample is processed; after obtaining the processing result, the execution time corresponding to each computing subgraph and the output tensor information are obtained; for any two computing subgraphs that are adjacent in the execution order, the execution processors corresponding to the two adjacent computing subgraphs are determined, and if the execution processors are different, the inference delay in the first execution information associated with the current computing subgraph is determined based on the output tensor information corresponding to the current computing subgraph and the delay coefficient.
[0101] In practical applications, after processing the current training sample, the processing result of the static computation graph under the current heterogeneous structure can be obtained, that is, the execution time of each computation subgraph in the current training sample on the execution processor in the server. It should be noted that in the process of processing the static computation graph based on the heterogeneous structure, the computation subgraph is copied between different execution processors, such as Figure 1 From the first execution processor to the second execution processor, in this case, it is necessary to consider not only the calculation sub Figure 1 The execution time on the second execution processor also needs to take into account the computation time. Figure 1 The inference delay from the first execution processor to the second execution processor is the copying time. Figure 2 and calculator Figure 3 For example, the calculator Figure 2 The output tensor of is the operator Figure 3 The input tensor of Figure 3 The output tensor does not need to be copied from the first execution processor to the second execution processor, indicating that the operator does not need to be considered. Figure 3 The inference latency of the input tensor from the first execution processor to the second execution processor. For this case where the inference latency of the computation subgraph does not need to be considered, it is only necessary to calculate the execution time of the computation subgraph on the deployed execution processor.
[0102] Specifically, although the static computation graph is divided into multiple computation subgraphs in this technical solution, there is a corresponding processing order in the actual processing of each computation subgraph. For example, the static computation graph includes 3 computation subgraphs, namely, Figure 1 , calculator Figure 2 and calculator Figure 3 , calculator Figure 1 The input tensor of is the tensor of the input data corresponding to the input data of the static computation graph. Figure 1 The output tensor of is the operator Figure 2 The input tensor of the operator Figure 2 The output tensor of is the operator Figure 3 When calculating the first execution information corresponding to each computational subgraph, it can be determined whether the inference delay of the corresponding computational subgraph needs to be considered according to whether two adjacent computational subgraphs are deployed on the same execution processor.
[0103] Specifically, if the calculator Figure 2 With calculator Figure 1 are deployed on the same execution processor, it means that the computation Figure 1 The output tensor of can be directly used as a computation subroutine on the deployed execution processor Figure 2 The input tensor of Figure 2 The corresponding first execution information does not need to consider the inference delay, only the calculation sub Figure 2 The execution time on the deployed execution processor is sufficient.
[0104] If the calculator Figure 2 Deployed on the first execution processor, the computation Figure 3 Deployed on the second execution processor, it means that after obtaining the computation sub Figure 2 After the output tensor of Figure 2 The output tensor is copied to the second execution processor and then used as the computation sub Figure 3 Therefore, in determining the operator Figure 2 When the corresponding first execution information is generated, not only the calculation sub Figure 2 The execution time on the first execution processor also needs to take into account the computation time of the subprocessor. Figure 2 The inference latency of the output tensor of is copied from the first execution processor to the second execution processor according to the computation sub Figure 2 Execution time and computation time on the first execution processor Figure 2 The corresponding inference delay is obtained with the operator Figure 2 The corresponding total execution time. Figure 2 The corresponding inference delay can be calculated based on the calculation sub Figure 2 The output tensor information of and the corresponding delay coefficient are multiplied to obtain the operator Figure 2 Corresponding inference latency. For example, according to the calculation sub Figure 3 The product of the tensor size of the output tensor and the delay coefficient corresponding to the tensor size is obtained by computing the sub Figure 2 The corresponding inference latency.
[0105] Here we take a specific example to explain the determination of the calculation sub Figure 2 The inference latency of Figure 2 It includes at least one operator and the input and output tensor information of the operator, and then the operator can be estimated based on the size information of the input and output tensors of all operators. Figure 2 The size of the corresponding input and output tensors. Since the tensor size and the delay coefficient usually have a linear relationship, according to the calculation sub Figure 2 The size of the output tensor can determine the operator Figure 2 The corresponding delay coefficient is then calculated based on the Figure 2 The product of the size of the output tensor and the corresponding delay coefficient can be obtained by computing the sub Figure 2 The corresponding inference latency.
[0106] It should be noted that when the output tensor of the computation subgraph needs to be copied from the first execution processor to the second execution processor, for example, Figure 2 and calculator Figure 3 , calculator Figure 2 The output tensor of is the operator Figure 3 The input tensor of the operator Figure 2 Executed on the first execution processor, the computation Figure 3 When calculating the execution time corresponding to the computation subgraph, the inference delay of the computation subgraph is considered. There are two ways to calculate it. One is to merge the inference delay into the computation subgraph. Figure 2 The corresponding execution time, that is, the calculation sub Figure 2 The corresponding execution time is the sum of the inference time on the first execution processor and the inference delay of copying the output tensor to the second execution processor. Figure 3 The corresponding execution time, that is, the calculation sub Figure 3 The corresponding execution time is the calculation sub Figure 3 Inference time and computation time on the second execution processor Figure 3 Receiving calculator Figure 2 The sum of the inference delays corresponding to the output tensors of . The specific merging method of inference delays is not limited in this technical solution, and only the first calculation method is used as an example for illustration.
[0107] S205: Determine the execution information corresponding to the execution cycle based on all the first execution information corresponding to each training sample.
[0108] Based on the above example, each execution cycle includes 100 iterations, and 100 training samples are included in the execution cycle. For each iteration, the first execution information corresponding to the current training sample will be recorded after the iteration is completed. Further, based on similar operations, the execution information corresponding to the current execution cycle can be obtained according to the first execution information of each training sample in the execution cycle.
[0109] Specifically, based on all the first execution information corresponding to each training sample, the execution information corresponding to the execution cycle is determined, including: for each training sample, according to the execution information corresponding to when all computing subgraphs process the training sample, the total execution time of the training sample is determined; and the minimum total execution time is used as the execution information of the execution cycle.
[0110] The total execution time refers to the minimum sum of the execution times of all computational subgraphs in the training sample on the deployed execution processors.
[0111] In practical applications, for each computational subgraph in a static computational graph, some computational subgraphs can be executed on any execution processor, but the execution time on different execution processors is different. If the copying problem of the input tensors of each computational subgraph is involved, the inference delay corresponding to the computational subgraph must also be considered.
[0112] Take one of the training samples as an example, and continue to refer to Figure 4 , the training sample includes 10 computational subgraphs. If each computational subgraph can be executed by any execution processor, the execution time of each computational subgraph on each execution processor is determined and recorded respectively. Furthermore, the deployment of these 10 computational subgraphs on different execution processors is arranged and combined, and several arrangements and combinations can be obtained when the first computational subgraph to the last computational subgraph is processed based on at least two execution processors. For example, there are a total of 5 arrangements and combinations of the deployment of the execution processors from the first computational subgraph to the last computational subgraph. According to the first execution information of each computational subgraph in the training sample on the corresponding execution processor, the 5 execution times corresponding to the training sample in this iteration are obtained, and the minimum execution time among these 5 execution times is used as the total execution time corresponding to the training sample.
[0113] Furthermore, for the 100 iterations within this execution cycle, 100 total execution times corresponding to the training samples in these 100 iterations are obtained in a similar way, and the smallest total execution time among these 100 total execution times is used as the execution information corresponding to the execution cycle (i.e., the inference time of the computational subgraph on the CPU and GPU and the inference time of the overall static graph are calculated respectively).
[0114] S206: modifying the execution processor corresponding to the computational subgraph based on the mutation factor, so that the updated heterogeneous structure processes the training samples of the next execution cycle to obtain the execution information of the next execution cycle.
[0115] Based on the above example, after processing each computational subgraph in the training sample on different execution processors, the execution processor adapted to each computational subgraph can be preliminarily obtained. In order to make the matching result of the execution processor of each computational subgraph more accurate and appropriate, take one of the computational subgraphs as an example, and continue to refer to Figure 4, the initial mutation factor corresponding to the computation subgraph is 0.5, which indicates that the computation subgraph can be executed on any execution processor. As the configuration of the execution processor of the computation subgraph is optimized, it is found that the execution time of the computation subgraph on the first execution processor is always shorter than that of the second execution processor, which indicates that the computation subgraph is more suitable for the first execution processor. After the computation subgraph is deployed on the first execution processor, the mutation factor corresponding to the computation subgraph is reduced, or after the computation subgraph is deployed on the second execution processor, the mutation factor corresponding to the computation subgraph is increased, so as to finally fix the computation subgraph on the first execution processor for execution.
[0116] On this basis, after each execution cycle, the computing architecture identifier and mutation factor of each computing subgraph are updated to obtain the heterogeneous structure corresponding to the reasoning task under the execution cycle, and the training samples of the next execution cycle are processed based on the updated structure to obtain the execution information of the next execution cycle.
[0117] S207 . Determine a heterogeneous structure of a next execution cycle of a later execution cycle of the two adjacent execution cycles based on execution information of the two adjacent execution cycles.
[0118] Based on the above example, the execution information of each execution cycle is calculated respectively, and the heterogeneous structure of the next execution cycle adjacent to the subsequent execution cycle is determined by comparing adjacent execution information.
[0119] Specifically, the execution information includes the total execution time corresponding to the execution cycle. Based on the execution information of two adjacent execution cycles, the heterogeneous structure of the next execution cycle of the latter execution cycle between the two adjacent execution cycles is determined, including: if the total execution time of the previous execution cycle between the two adjacent execution cycles is less than the total execution time of the latter execution cycle, the latter execution cycle is removed, and the execution processor corresponding to the calculation subgraph is changed based on the mutation factor, so that the updated heterogeneous structure processes the training samples of the next execution cycle to obtain the execution information of the next execution cycle.
[0120] For example, see Figure 4 , respectively obtain the execution information corresponding to the first execution cycle and the second execution cycle. If the total execution time corresponding to the first execution cycle is less than the total execution time corresponding to the second execution cycle, remove the second execution cycle, that is, continue to use the heterogeneous structure determined in the first execution cycle as the heterogeneous structure of the third execution cycle adjacent to the second execution cycle.
[0121] S208. Determine a target execution processor for each computation subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computation subgraph.
[0122] In practical applications, based on the execution information of each execution cycle, the target execution processor of each computational subgraph in the heterogeneous structure is determined, including: taking the heterogeneous structure corresponding to the minimum total execution time in the execution information as the target heterogeneous structure; and obtaining the target execution processor corresponding to each computational subgraph in the target heterogeneous structure.
[0123] Based on the above example, continue to see Figure 4 , based on similar operations, after completing an execution cycle of the reasoning task, continue to perform iterative optimization of the next execution cycle until 5 execution cycles are completed, and the target executor corresponding to each computational subgraph is determined according to the execution information corresponding to each computational subgraph recorded in each execution cycle, so as to obtain the target heterogeneous structure corresponding to the reasoning task according to the target executor of each computational subgraph, and then in the process of executing the target task, the target execution processor corresponding to each computational subgraph in the target heterogeneous structure is used for processing. In other words, according to whether the total execution time corresponding to the first execution cycle is less than the total execution time corresponding to the second execution cycle, a new round of mutation cycle can be started according to the updated one, until there is no update after N rounds, it is considered that the mutation result of the execution processor of the computational subgraph in the entire static computational graph has reached a convergence state, and the heterogeneous structure at this time is used as the target heterogeneous structure to continue to execute the target task.
[0124] The purpose of this setting is to update the heterogeneous structure used in subsequent execution cycles by comparing the total execution time corresponding to the processed execution cycles, so as to ensure that the heterogeneous architecture used in the execution cycle corresponding to the shortest total execution time among multiple execution cycles is finally used as the target heterogeneous architecture, and then process each computational subgraph in the target task of the same type as the reasoning task based on the target heterogeneous architecture, so as to achieve the optimal configuration of the execution processor in the target task processing process.
[0125] In an embodiment of the present invention, for at least one training sample, the current training sample is processed based on at least one computational subgraph in the heterogeneous structure to obtain the first execution information of each computational subgraph; based on all the first execution information corresponding to each training sample, the execution information corresponding to the execution cycle is determined; based on the mutation factor, the execution processor corresponding to the computational subgraph is changed so that the updated heterogeneous structure processes the training sample of the next execution cycle to obtain the execution information of the next execution cycle; based on the execution information of two adjacent execution cycles, the heterogeneous structure of the next execution cycle of the latter execution cycle of the two adjacent execution cycles is determined. In this technical solution, in order to determine the optimal target heterogeneous structure corresponding to the reasoning task, it is necessary to perform multiple iterative optimizations on the execution processors corresponding to the multiple computational subgraphs associated with the reasoning task. Specifically, by setting at least one execution cycle corresponding to the reasoning task, each execution cycle includes at least one iteration, and each iterative process includes at least one iterative optimization process, and the training data in each iterative optimization process is used as a training sample for training. Furthermore, after each iteration of optimization, the first execution information corresponding to each training sample is recorded, and the execution information corresponding to the execution cycle is determined based on all the first execution information in the execution cycle, and then the target heterogeneous structure corresponding to the reasoning task is finally determined based on the execution information corresponding to all the execution cycles. This solves the problem in the prior art that developers are not familiar with the underlying operators in the deep learning network architecture and the heterogeneous copy information of the input and output data in the static calculation graph, and are unable to configure a reasonable heterogeneous architecture for the static calculation graph corresponding to the reasoning task, resulting in unreasonable resource use of the execution processor in the server. This realizes the automatic configuration of the most reasonable heterogeneous architecture for the static calculation graph corresponding to the reasoning task, so that the reasoning task is jointly executed based on multiple execution processors in the server, thereby maximizing the resource advantages of different execution processors in the server.
[0126] Figure 5 A schematic diagram of the structure of the device for executing the execution processor provided in the embodiment of the present application. Figure 5 As shown, the device that the execution processor cooperates with to execute includes: a file loading module 301, a subgraph partitioning module 302, a heterogeneous structure determination module 303, an execution information determination module 304 and a target task processing module 305.
[0127] The file loading module 301 is used to load the network structure file of the static calculation graph corresponding to the reasoning task; wherein the reasoning task is the task of how each neural network layer in the neural network processes the input data to obtain the output result that meets the preset conditions, and the static calculation graph includes multiple operators, and the operators are associated with the neural network layers, and are used to limit the processing information of the input data;
[0128] A subgraph partitioning module 302 is used to partition the static computation graph into at least two computation subgraphs according to the network structure file and the operator cutting constraint condition, and determine a subgraph file corresponding to the computation subgraph; wherein the subgraph file includes a computation architecture identifier and input and output tensors of the computation subgraph;
[0129] The heterogeneous structure determination module 303 is used to configure a mutation factor for the computation subgraph according to the computation architecture identifier and input and output tensors of each computation subgraph, so as to obtain a heterogeneous structure corresponding to the reasoning task; wherein the mutation factor is used to adjust the execution processor corresponding to the computation subgraph, and the execution processor includes a central processing unit or a graphics processing unit;
[0130] An execution information determination module 304 is used to determine, for at least one training sample in at least one execution cycle, execution information of each computational subgraph based on at least one training sample, a heterogeneous structure, and a mutation factor corresponding to the computational subgraph in the heterogeneous structure; wherein the execution information includes inference time and inference latency, and the inference latency is information on the time required to copy an output tensor obtained by a current execution processor to another execution processor;
[0131] The target task processing module 305 is used to determine the target execution processor of each computation subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computation subgraph.
[0132] In an embodiment of the present invention, by loading a network structure file of a static computational graph corresponding to an inference task; dividing the static computational graph into at least two computational subgraphs according to the network structure file and the operator cutting constraint, and determining the subgraph file corresponding to the computational subgraph; configuring a mutation factor for the computational subgraph according to the computational architecture identifier and the input and output tensors of each computational subgraph, and obtaining a heterogeneous structure corresponding to the inference task; for at least one training sample in at least one execution cycle, determining the execution information of each computational subgraph based on at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure; determining the target execution processor of each computational subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computational subgraph. In the present technical solution, at least two computational subgraphs can be obtained by loading a static computational graph corresponding to an inference task and performing subgraph cutting on the static computational graph, and on this basis, each computational subgraph can be configured on each execution processor for simulation operation. Among them, in the process of executing each computational subgraph and optimizing the configuration of the processor, the mutation of the computational subgraph on different execution processors can be controlled by setting the computational architecture identifier and mutation factor corresponding to each computational subgraph, so as to obtain the execution information of the computational subgraph on each execution processor. After the configuration of the execution processor corresponding to each computational subgraph is iteratively optimized for many times, the target execution processor corresponding to each computational subgraph is obtained according to the execution information corresponding to each computational subgraph in each iterative optimization process, that is, the execution processor with the shortest processing time for processing the computational subgraph is used as the target execution processor, and then the target heterogeneous structure corresponding to the reasoning task is determined according to the target execution processors corresponding to all the computational subgraphs, so as to realize the optimal resource configuration of at least two execution processors in the server during the execution of the reasoning task, so as to directly process the computational subgraph in the target task based on the determined target heterogeneous structure when processing the target task of the same type as the reasoning task. The present invention solves the problem in the prior art that, in the case where multiple execution processors that can be used to execute reasoning tasks are deployed in a server, when the overall reasoning task is reasoned and processed only based on a single execution processor, there is a waste of resources of the execution processor in the server. By sub-cutting the static calculation graph corresponding to the reasoning task and allocating at least one obtained calculation sub-graph to a reasonable execution processor for processing, the reasoning task is processed jointly based on at least two execution processors in the server. In this way, in the process of executing the reasoning task, multiple execution processors in the server are rationally configured, the resource advantages of each execution processor are fully utilized, and the processing efficiency of the reasoning task is improved.
[0133] In some embodiments, optionally, the file loading module includes: a computation graph generation submodule, configured to generate a static computation graph based on the acquired reasoning task;
[0134] The file loading submodule is used to load the network structure file of the static computation graph when it is determined that the static computation graph has heterogeneous verification conditions.
[0135] In some embodiments, optionally, a subgraph partitioning module is used to partition the network structure file according to the execution order corresponding to each operator in the static computation graph, the corresponding input and output tensors, the execution order based on the operator processing of the input data, whether the operator is located on a branch node on the static computation graph, and the operator quantity threshold corresponding to each computation subgraph, so as to obtain at least two computation subgraphs.
[0136] In some embodiments, optionally, a subgraph partitioning module is used to determine the file content of a subgraph file corresponding to a computational subgraph based on a computational architecture identifier and input and output tensors of at least one operator in each computational subgraph.
[0137] In some embodiments, optionally, the heterogeneous structure determination module includes: a first mutation factor setting submodule, configured to configure a first mutation factor for the computation subgraph when the computation architecture identifier of the computation subgraph is a first identifier;
[0138] A second mutation factor setting submodule, used to configure a second mutation factor for the computation subgraph when the computation architecture identifier of the computation subgraph is a second identifier;
[0139] The heterogeneous structure determination submodule is used to determine the heterogeneous structure corresponding to the reasoning task based on the mutation factor, input and output tensors corresponding to each computational subgraph, and the processing order corresponding to each computational subgraph.
[0140] In some embodiments, optionally, the execution information determination module includes: a first execution information determination submodule, configured to process the current training sample based on at least one computation subgraph in the heterogeneous structure for at least one training sample to obtain first execution information of each computation subgraph;
[0141] An execution information determination submodule, used to determine the execution information corresponding to the execution cycle based on all the first execution information corresponding to each training sample;
[0142] The next execution cycle processing submodule is used to change the execution processor corresponding to the computation subgraph based on the mutation factor, so that the updated heterogeneous structure processes the training samples of the next execution cycle to obtain the execution information of the next execution cycle;
[0143] The heterogeneous structure determination submodule is used to determine the heterogeneous structure of the next execution cycle of the latter execution cycle of the two adjacent execution cycles based on the execution information of the two adjacent execution cycles.
[0144] In some embodiments, optionally, the first execution information determination submodule includes: a current training sample processing unit, configured to process the current training sample based on the execution processor deployed by each computation subgraph in the heterogeneous structure;
[0145] An information acquisition unit is used to obtain the execution time corresponding to each computational subgraph and output tensor information after obtaining the processing result;
[0146] The inference delay determination unit is used to determine the execution processors corresponding to any two adjacent computational subgraphs in the execution order. If the execution processors are different, the inference delay in the first execution information associated with the current computational subgraph is determined based on the output tensor information and the delay coefficient corresponding to the current computational subgraph.
[0147] In some embodiments, optionally, the execution information determination submodule includes: a total execution time determination unit, configured to determine, for each training sample, the total execution time of the training sample according to the execution information corresponding to the processing of the training sample by all computational subgraphs;
[0148] An execution information determination unit is used to use the minimum total execution time as the execution information of the execution cycle. In some embodiments, optionally, the heterogeneous structure determination submodule includes: a heterogeneous structure determination unit, which is used for the execution information to include the total execution time. If the total execution time of the previous execution cycle in two adjacent execution cycles is less than the total execution time of the next execution cycle, the next execution is removed, and the execution processor corresponding to the computational subgraph is changed based on the mutation factor, so that the updated heterogeneous structure processes the training samples of the next execution cycle to obtain the execution information of the next execution cycle.
[0149] In some embodiments, optionally, the target task processing module includes: a target heterogeneous structure determination submodule, configured to use the heterogeneous structure corresponding to the minimum total execution time in the execution information as the target heterogeneous structure;
[0150] The target processor acquisition submodule is used to obtain the target execution processor corresponding to each computational subgraph in the target heterogeneous structure.
[0151] In some embodiments, optionally, the device for executing in cooperation with the processor further includes: a task receiving module, configured to receive a target task;
[0152] The result output module is used to process the target task based on the target execution processor corresponding to each computational subgraph and output the target result of the target task.
[0153] The device for executing processors in cooperation with the execution provided in the embodiments of the present application can be used to execute the technical solution of the method for executing processors in cooperation with the execution in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.
[0154] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the file loading module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.
[0155] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .
[0156] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the scheme in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0157] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.
[0158] The transceiver 121 may be used to send a target processing result corresponding to the service request.
[0159] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.
[0160] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.
[0161] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the method of cooperating with the execution processor in the above embodiment.
[0162] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the method of executing the processor in the above embodiment.
[0163] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the method for executing the processor in cooperation with the execution provided in any embodiment of the present application.
[0164] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a method as provided in any embodiment of the present application. . .
[0165] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0166] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A method for executing a processor in cooperation with another processor, characterized in that: include: Loading a network structure file of a static computation graph corresponding to an inference task; wherein the inference task is a task of how each neural network layer in a neural network processes input data to obtain an output result that meets preset conditions, and the static computation graph includes a plurality of operators, and the operators are associated with the neural network layers and are used to limit processing information of the input data; According to the network structure file and the operator cutting constraint condition, the static computation graph is divided into at least two computation subgraphs, and a subgraph file corresponding to the computation subgraph is determined; wherein the subgraph file includes a computation architecture identifier and input and output tensors of the computation subgraph; According to the computing architecture identifier and input and output tensors of each computing subgraph, a mutation factor is configured for the computing subgraph to obtain a heterogeneous structure corresponding to the reasoning task; wherein the mutation factor is used to adjust the execution processor corresponding to the computing subgraph, and the execution processor includes a central processing unit or a graphics processing unit; For at least one training sample in at least one execution cycle, based on the at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure, determining execution information of each computational subgraph; wherein the execution information includes inference time and inference latency, and the inference latency is the time required to copy the output tensor obtained by the current execution processor to another execution processor; Based on the execution information of each execution cycle, a target execution processor of each computation subgraph in the heterogeneous structure is determined, so as to process the received target task based on the target execution processor corresponding to each computation subgraph.
2. The method according to claim 1, characterized in that The loading of the network structure file of the static computation graph corresponding to the reasoning task includes: Generate a static computation graph based on the acquired reasoning task; When it is determined that the static computation graph has a heterogeneous verification condition, a network structure file of the static computation graph is loaded.
3. The method according to claim 1, characterized in that The step of dividing the static computation graph into at least two computation subgraphs according to the network structure file and the operator cutting constraint condition includes: According to the execution order corresponding to each operator in the static calculation graph, the corresponding input and output tensors, the execution order based on the operator processing the input data, whether the operator is on the branch node on the static calculation graph, and the operator quantity threshold corresponding to each calculation subgraph, the network structure file is split to obtain at least two calculation subgraphs.
4. The method according to claim 1, characterized in that: The determining the subgraph file corresponding to the computation subgraph includes: According to the computing architecture identifier and input and output tensors of at least one operator in each computing subgraph, the file content of the subgraph file corresponding to the computing subgraph is determined.
5. The method according to claim 1, characterized in that The configuring a mutation factor for each computational subgraph according to the computational architecture identifier and input and output tensors of each computational subgraph to obtain a heterogeneous structure corresponding to the reasoning task includes: When the computing architecture identifier of the computing subgraph is a first identifier, configuring a first mutation factor for the computing subgraph; When the computing architecture identifier of the computing subgraph is a second identifier, configuring a second mutation factor for the computing subgraph; Based on the mutation factor corresponding to each computational subgraph, the input and output tensors, and the processing order corresponding to each computational subgraph, a heterogeneous structure corresponding to the reasoning task is determined.
6. The method according to claim 1, characterized in that The determining, based on the at least one training sample, the heterogeneous structure, and the mutation factor corresponding to the computational subgraph in the heterogeneous structure, execution information of each computational subgraph includes: For the at least one training sample, processing the current training sample based on at least one computation subgraph in the heterogeneous structure to obtain first execution information of each computation subgraph; Determining the execution information corresponding to the execution cycle based on all first execution information corresponding to each training sample; Based on the mutation factor, the execution processor corresponding to the computational subgraph is changed so that the updated heterogeneous structure processes the training samples of the next execution cycle to obtain the execution information of the next execution cycle; Based on the execution information of two adjacent execution cycles, a heterogeneous structure of a next execution cycle of a later execution cycle of the two adjacent execution cycles is determined.
7. The method according to claim 6, characterized in that The processing of the current training sample based on at least one computation subgraph in the heterogeneous structure to obtain first execution information of each computation subgraph includes: Processing the current training sample based on the execution processor deployed by each computational subgraph in the heterogeneous structure; After obtaining the processing results, obtain the execution time corresponding to each computational subgraph and the output tensor information; For any two computational subgraphs that are adjacent in execution order, determine the execution processors corresponding to the two adjacent computational subgraphs. If the execution processors are different, determine the inference delay in the first execution information associated with the current computational subgraph based on the output tensor information and delay coefficient corresponding to the current computational subgraph.
8. The method according to claim 6, characterized in that The determining, based on all first execution information corresponding to each training sample, execution information corresponding to the execution cycle includes: For each training sample, determine the total execution time of the training sample according to the execution information corresponding to the processing of the training sample by all computational subgraphs; The minimum total execution time is used as the execution information of the execution cycle.
9. The method according to claim 8, characterized in that The execution information includes a total execution time. The determining of the heterogeneous structure of the next execution cycle of the latter execution cycle of the two adjacent execution cycles based on the execution information of the two adjacent execution cycles includes: If the total execution time of the previous execution cycle of two adjacent execution cycles is less than the total execution time of the subsequent execution cycle, the subsequent execution cycle is removed, and the execution processor corresponding to the calculation subgraph is changed based on the mutation factor, so that the updated heterogeneous structure processes the training samples of the next execution cycle to obtain the execution information of the next execution cycle.
10. The method according to claim 9, characterized in that The step of determining a target execution processor for each computational subgraph in the heterogeneous structure based on the execution information of each execution cycle includes: The heterogeneous structure corresponding to the minimum total execution time in the execution information is used as the target heterogeneous structure; Obtain a target execution processor corresponding to each computational subgraph in the target heterogeneous structure.
11. The method according to claim 1, characterized in that: After obtaining the target execution processor corresponding to each computation subgraph, the method further includes: Receive target tasks; The target task is processed based on the target execution processor corresponding to each computational subgraph, and a target result of the target task is output.
12. A device for executing a processor in cooperation with an execution processor, characterized in that: include: A file loading module is used to load a network structure file of a static computation graph corresponding to an inference task; wherein the inference task is a task of how each neural network layer in a neural network processes input data to obtain an output result that meets preset conditions, and the static computation graph includes a plurality of operators, and the operators are associated with the neural network layers and are used to limit processing information of the input data; A subgraph partitioning module, used to partition the static computation graph into at least two computation subgraphs according to the network structure file and the operator cutting constraint condition, and determine a subgraph file corresponding to the computation subgraph; wherein the subgraph file includes a computation architecture identifier and input and output tensors of the computation subgraph; A heterogeneous structure determination module, configured to configure a mutation factor for each computational subgraph according to the computational architecture identifier and input and output tensors of the computational subgraph, so as to obtain a heterogeneous structure corresponding to the reasoning task; wherein the mutation factor is used to adjust an execution processor corresponding to the computational subgraph, the execution processor including a central processing unit or a graphics processing unit; An execution information determination module is used to determine, for at least one training sample in at least one execution cycle, execution information of each computational subgraph based on the at least one training sample, the heterogeneous structure, and a mutation factor corresponding to the computational subgraph in the heterogeneous structure; wherein the execution information includes inference time and inference delay, and the inference delay is the time information required to copy the output tensor obtained by the current execution processor to another execution processor; The target task processing module is used to determine the target execution processor of each computing subgraph in the heterogeneous structure based on the execution information of each execution cycle, so as to process the received target task based on the target execution processor corresponding to each computing subgraph.
13. An electronic device that is executed in cooperation with an execution processor, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for executing the computer program in cooperation with the execution processor as described in any one of claims 1 to 11.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for executing the program in cooperation with the processor as described in any one of claims 1 to 11 is implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method for executing in cooperation with the processor as described in any one of claims 1 to 11.
Citation Information
Cited By
Method, device and equipment for determining operator execution provider and storage medium
CN122819503A
