Operator optimization method and apparatus, and storage medium and electronic device

Abstract

Provided in the present disclosure are an operator optimization method and apparatus, and a storage medium and an electronic device. The operator optimization method provided in the present disclosure comprises: acquiring a target neural network model, and determining a computational graph of the target neural network model; for each operator in the computational graph, determining a search space including all feasible solutions for the operator; selecting, from the search space, several feasible solutions as candidate solutions, determining evaluation values of the candidate solutions, and using a candidate solution having the highest evaluation value as a tentative solution; determining a running time during which target hardware runs the tentative solution, and increasing the number of iterations; when the running time is less than the current optimal time or there is no current optimal time, determining the running time to be the current optimal time, and determining the tentative solution to be the current optimal solution; when the number of iterations is less than a specified number of times, reselecting, from the search space of the operator, a specified number of candidate solutions that have not been selected; and when the number of iterations is not less than the specified number of times, determining the current optimal solution to be an optimal solution for the operator.

Description

Operator optimization method, device, storage medium and electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310941263.1 filed on July 28, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to an operator optimization method, device, storage medium, and electronic device. Background Art

[0004] In recent years, with the continuous development of deep learning technology, the number of neural network model parameters has increased, and the structure has become more complex, the requirements for model deployment have also become increasingly stringent. However, building and tuning high-performance computing libraries requires a huge amount of engineering effort, and the ever-changing structure of neural networks undoubtedly creates tremendous engineering pressure.

[0005] To address these issues, a solution using compilation technology to map the computational graphs corresponding to neural network models to different hardware and generate hardware executable code has achieved significant results. In this process, optimizing the operators in the computational graph to improve the compatibility of the neural network model with the hardware is a crucial step.

[0006] Existing methods for operator optimization test all possible implementations of the operator in hardware, selecting the optimal solution with the shortest runtime. However, operator optimization in these approaches is often very time-consuming and costly.

[0007] Summary of the Invention

[0008] The present disclosure provides a method, an apparatus, a storage medium, and an electronic device to solve the above-mentioned problems existing in the related art.

[0009] The present disclosure provides an operator optimization method, comprising:

[0010] Obtaining a target neural network model and determining a computational graph of the target neural network model;

[0011] For each operator in the computation graph, determining a search space for the operator, wherein the search space contains all feasible solutions for the operator;

[0012] Selecting a specified number of feasible solutions as candidate solutions in the search space of the operator, determining an evaluation value of each candidate solution, and selecting the candidate solution with the highest evaluation value as the pending solution;

[0013] Determine the runtime of the target hardware to run the undetermined solution and increase the number of iterations;

[0014] When the running time is less than the current optimal time or there is no current optimal time, the running time is determined as the current optimal time, and the undetermined solution is determined as the current optimal solution;

[0015] When the number of iterations is less than the specified number, reselecting a specified number of unselected candidate solutions in the search space of the operator;

[0016] When the number of iterations is not less than the specified number, the current optimal solution is determined as the optimal solution of the operator.

[0017] Optionally, select a specified number of candidate solutions in the search space of the operator, including:

[0018] Select the specified solution from the feasible solutions of the operator as the candidate solution;

[0019] A neighborhood solution of the designated solution is determined among feasible solutions of the operator, and the neighborhood solution is re-determined as the designated solution until the number of determined candidate solutions reaches a designated number.

[0020] Optionally, determining an evaluation value of each candidate solution includes:

[0021] For each candidate solution, input the candidate solution into a pre-trained evaluation model to obtain an evaluation result of the candidate solution output by the evaluation model;

[0022] An evaluation value of the candidate solution is determined according to the evaluation result.

[0023] Optionally, the evaluation model includes several prediction models; the candidate solution is input into a pre-trained evaluation model to obtain an evaluation result of the candidate solution output by the evaluation model, including:

[0024] Input the candidate solution into each prediction model in the pre-trained evaluation model to obtain the predicted running time output by each prediction model;

[0025] The evaluation result of the candidate solution is determined according to each prediction running time.

[0026] Optionally, the evaluation result includes an average running time and a standard deviation; determining the evaluation result of the candidate solution according to each predicted running time includes:

[0027] The average running time and standard deviation of the candidate solution are determined based on the predicted running times.

[0028] Optionally, determining an evaluation value of the candidate solution according to the evaluation result includes:

[0029] An evaluation value of the candidate solution is determined based on the evaluation result and the current optimal time.

[0030] Optionally, pre-trained evaluation models, including:

[0031] Get sample operator;

[0032] Determining a sample feasible solution of the sample operator and a real running time of the sample feasible solution in target hardware;

[0033] For each prediction model, the sample feasible solution is input into the prediction model to obtain the prediction running time to be optimized output by the prediction model, wherein the sample feasible solution input into each prediction model belongs to a different sample operator;

[0034] The prediction model is trained with the optimization goal of minimizing the difference between the predicted running time of the sample to be optimized and the actual running time of the sample feasible solution input into the prediction model in the target hardware.

[0035] The present disclosure provides an operator optimization device, comprising:

[0036] An acquisition module, configured to acquire a target neural network model and determine a computational graph of the target neural network model;

[0037] a determination module, configured to determine, for each operator in the computation graph, a search space for the operator, wherein the search space contains all feasible solutions for the operator;

[0038] A selection module is used to select a specified number of feasible solutions as candidate solutions in the search space of the operator, determine the evaluation value of each candidate solution, and select the candidate solution with the highest evaluation value as the undetermined solution;

[0039] Adding a module for determining the runtime of the target hardware to run the undetermined solution and increasing the number of iterations;

[0040] an updating module, configured to, when the running time is less than the current optimal time or the current optimal time does not exist, determine the running time as the current optimal time and determine the undetermined solution as the current optimal solution;

[0041] An iteration module, configured to reselect a specified number of unselected candidate solutions in the search space of the operator when the number of iterations is less than a specified number;

[0042] The optimization module is configured to determine the current optimal solution as the optimal solution of the operator when the number of iterations is not less than the specified number.

[0043] The present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned operator optimization method is implemented.

[0044] The present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned operator optimization method when executing the program.

[0045] At least one of the above technical solutions adopted in the present disclosure can achieve the following beneficial effects:

[0046] In the operator optimization method provided by the present disclosure, a target neural network model is obtained, and a computational graph of the target neural network model is determined; for each operator in the computational graph, a search space containing all feasible solutions of the operator is determined; several feasible solutions are selected in the search space as candidate solutions, an evaluation value of each candidate solution is determined, and the solution with the highest evaluation value is selected as the pending solution; the running time of the target hardware to run the pending solution is determined, and the number of iterations is increased; when the running time is less than the current optimal time or there is no current optimal time, the running time is determined as the current optimal time, and the pending solution is determined as the current optimal solution; when the number of iterations is less than a specified number, a specified number of unselected candidate solutions are reselected in the search space of the operator; when the number of iterations is not less than a specified number, the current optimal solution is determined as the optimal solution of the operator.

[0047] When optimizing operators in the computational graph of a target neural network model using the operator optimization method provided by this disclosure, a locally optimal solution can be found by iterating through the operator's search space, selecting a specified number of candidate solutions without duplication in each iteration. Finally, a globally optimal solution is found among the multiple locally optimal solutions, which serves as the optimal solution for the operator. Compared to traditional methods, the operator optimization method provided by this disclosure, based on the concept of Bayesian optimization, searches for potential optimal solutions through pre-evaluation. This method can significantly reduce the number of times different feasible solutions of the operator are run on the target hardware, significantly reducing the time cost and overhead of operator optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0049] FIG1 is a schematic flow chart of an operator optimization method according to an embodiment of the present disclosure;

[0050] FIG2 is a schematic diagram of an operator optimization device according to an embodiment of the present disclosure;

[0051] FIG. 3 is a schematic diagram of an electronic device corresponding to FIG. 1 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Many neural network models can now be exported as computational graphs. For example, models trained using deep learning frameworks like TensorFlow and PyTorch can be exported as computational graphs. During deployment, the operators in the computational graphs are mapped to the computational libraries provided by the hardware vendor for execution.

[0053] Currently, highly efficient deep learning compilers typically involve the following process: first, they import the computational graph corresponding to the neural network model as a high-level intermediate representation (IR), perform graph-level optimizations, then convert the optimized high-level IR into a low-level IR, perform operator-level optimizations, and finally perform code generation to convert the low-level IR into executable code for the target hardware. Deep learning compilers such as TVM and Halide, for example, employ a similar process.

[0054] However, at present, the process of optimizing operators in the above process is still immature, time-consuming and costly. To solve the above problems, the present disclosure provides a more efficient operator optimization method.

[0055] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0056] The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0057] FIG1 is a flow chart of an operator optimization method according to an embodiment of the present disclosure, which specifically includes the following steps S100 to S112 .

[0058] S100: Obtain a target neural network model and determine a computational graph of the target neural network model.

[0059] All steps in the operator optimization method provided in the present disclosure can be implemented by any electronic device with computing capabilities, such as a terminal, a server, and the like.

[0060] The operator optimization method provided in this disclosure is used to optimize the operators in the computation graph derived from a neural network model with high efficiency, so as to improve the speed of the operators in the neural network model when running on hardware. Based on this, in this step, the target neural network model can be first obtained, and after obtaining the target neural network model, the computation graph of the target neural network model can be determined.

[0061] A computation graph is a data structure used to represent computational processes. Like general graph data, it is composed of nodes and edges. In a computation graph derived from a neural network model, nodes represent network layers, with each node corresponding to a different network layer in the neural network model. Edges represent data flow, that is, the transfer of data between nodes (network layers). Each node contains several operators, each representing an operation, such as matrix multiplication. Different operators represent different operations, and the operator contained in each node is used to implement the function of the network layer corresponding to that node.

[0062] S102: For each operator in the computation graph, determine a search space for the operator, where the search space contains all feasible solutions for the operator.

[0063] In the computational graph of the neural model network determined in step S100, there will be several operators. The goal of this method is to ensure to the greatest extent possible that each operator can run on the hardware at the fastest speed, and the speed at which the operator runs on the hardware usually depends on the degree of adaptability of the operator's feasible solution to the hardware. Among them, a feasible solution of an operator is a feasible way to calculate the operator. Generally, an operator will have multiple different feasible solutions, and the degree of adaptability of each feasible solution to the hardware may be different. The higher the degree of adaptability of a feasible solution of an operator to the hardware, the faster the hardware runs the operator under the feasible solution; conversely, the lower the degree of adaptability of a feasible solution of an operator to the hardware, the slower the speed at which the hardware runs the operator under the feasible solution.

[0064] Therefore, in this step, we first determine the search space for each operator. An operator's search space includes all feasible solutions for that operator. Determining all feasible solutions for each operator allows us to find feasible solutions that are highly compatible with the hardware in subsequent steps.

[0065] More optimally, before determining the search space for each operator, identical operators can be prioritized for merging to further improve the efficiency of operator optimization. Typically, each node in a computation graph is likely to contain duplicate operators. For example, the same operator may be applied repeatedly to implement the functionality of the network layer corresponding to a node; or, in another example, the same operator may be used to implement the functionality of the network layer corresponding to different nodes. These situations are quite common in practical applications. On the other hand, the adaptability of the same operator on the same hardware, that is, its operating efficiency, remains constant, regardless of the node where the operator resides. Based on this, for an operator that appears multiple times in the computation graph, it only needs to be optimized once. Therefore, before determining the search space for each operator, identical operators in the computation graph can be merged to retain no more than one of each operator, and then the search space for each operator can be determined.

[0066] S104: Select a specified number of feasible solutions in the search space of the operator as candidate solutions, determine an evaluation value of each candidate solution, and select the candidate solution with the highest evaluation value as the pending solution.

[0067] For each operator, a specified number of feasible solutions are selected from the operator's search space and used as candidate solutions. Subsequently, the evaluation value of each candidate solution is determined, and the candidate with the highest evaluation value is selected as the pending solution. The specified number can be set according to specific needs.

[0068] The evaluation value of a candidate solution for an operator represents the speed at which the target hardware will execute the operator under that candidate solution. The higher the evaluation value of a candidate solution, the faster the target hardware will execute the corresponding operator under that candidate solution. It's important to note that the evaluation value is a prediction used in this method to assist in the execution of subsequent steps, and is not the actual result.

[0069] In the operator optimization method provided herein, steps S104 through S112 are executed in a loop. Until certain conditions are met, steps S104 through S112 must be executed repeatedly to find the optimal solution for the operator. Step S104 marks the beginning of the loop, and at the beginning of each round, a specified number of feasible solutions must be reselected as candidate solutions. Furthermore, to maximize the scope of the search space, the specified number of feasible solutions selected each time must have not been selected in the previous round.

[0070] More optimally, in the process of determining candidate solutions, based on the concept of Bayesian optimization, a neighborhood solution approach can be employed to ensure that the determined candidate solutions are relatively close to each other, thereby ensuring that the candidate solutions determined in each round are different as much as possible. Specifically, a designated solution can be selected from the feasible solutions of the operator as a candidate solution; the neighborhood solutions of the designated solution are determined from the feasible solutions of the operator, and the neighborhood solutions are re-determined as designated solutions until the number of determined candidate solutions reaches the specified number.

[0071] The neighborhood solutions of a specified solution are feasible solutions adjacent to the specified solution. For example, suppose a matrix multiplication operator has two different dimensional transformations. There are four possible transformations in the first dimension and 55 different possible transformations in the second dimension. Then, the operator has 4×55=220 different feasible solutions. Here, (x1, x2) is used to represent a feasible solution, where x1 represents one possible transformation in the first dimension and x2 represents one possible transformation in the second dimension. It can be imagined that in this example, 1≤x1≤4, 1≤x2≤55, and x1 and x2 are both integers. For any two feasible solutions of the same operator, if the two feasible solutions differ in only one dimension of transformation, then the two feasible solutions are neighbors of each other. For example, assuming the specified solution in the above example is (4, 20), then its neighborhood solutions can be (3, 20), (1, 20), (4, 25), (4, 10), and so on. As long as there is only one dimension in which two feasible solutions differ in transformation, then these two feasible solutions are neighboring solutions to each other.

[0072] In the above method, for each operator, a specified solution can be first selected from the feasible solutions of the operator, and the specified solution can be determined as one of the candidate solutions; then a neighborhood solution of the specified solution is determined, and the determined neighborhood solution replaces the original specified solution and is determined as the new specified solution, and the specified solution is continued to be determined as one of the candidate solutions... Repeat the above process until the number of determined candidate solutions reaches the specified number. It can be seen that the above process needs to be repeated many times until the number of determined candidate solutions reaches the specified number. In this process, the determined specified solution and each neighborhood solution will eventually become a candidate solution. The candidate solutions determined by the above method are relatively close in transformation mode, and it is easier to determine the local optimal pending solution; combined with the process of repeating multiple rounds of determining candidate solutions and pending solutions, the local optima are combined to more accurately obtain the global optimal feasible solution.

[0073] In one embodiment, various methods can be used to determine the evaluation value of each candidate solution. This disclosure provides one embodiment for reference. Specifically, for each candidate solution, the candidate solution can be input into a pre-trained evaluation model, and the evaluation result of the candidate solution output by the evaluation model can be obtained. Based on the evaluation result, the evaluation value of the candidate solution can be determined.

[0074] The above method evaluates each candidate solution through a pre-trained evaluation model. The specific structure and parameters of the evaluation model can be set according to needs, and the method of training the evaluation model can also change with the change of the structure of the evaluation model itself. The present disclosure does not impose specific restrictions on this. Similarly, the present disclosure provides a specific embodiment for reference. Specifically, the evaluation model may include several prediction models; when determining the evaluation value of a candidate solution, the candidate solution can be input into each prediction model in the pre-trained evaluation model to obtain the predicted running time output by each prediction model; the evaluation result of the candidate solution is determined according to each predicted running time.

[0075] In the above-mentioned specific embodiment, the evaluation model adopts the concept of random forest model and is composed of multiple different prediction models. The prediction models can be models with different structures and parameters, or models with the same structure but different parameters. This disclosure does not impose any specific restrictions on this. It only needs to ensure that for the same input, each model can produce different outputs.

[0076] Each prediction model performs the same function: given a candidate solution for an operator, it outputs the predicted runtime for that operator under that candidate solution. However, since each prediction model is unique, the results provided by each model will inevitably differ. In this case, the candidate solution's evaluation result is determined based on the predicted runtimes provided by each prediction model.

[0077] When multiple prediction models are used to form an evaluation model based on the random forest concept, the evaluation results of the evaluation models can also be correlated with the predicted runtimes obtained by the prediction models. Specifically, the evaluation results can include the average runtime and standard deviation. When determining the evaluation results of candidate solutions based on the predicted runtimes output by the prediction models, the average runtime and standard deviation of the candidate solutions can be determined based on the predicted runtimes.

[0078] The average running time may be the average of the predicted running times, and the standard deviation may be the standard deviation of the predicted running times. These two values ​​can more objectively and accurately reflect the running speed of an operator under a candidate solution on the hardware.

[0079] Furthermore, since the evaluation results of the candidate solutions themselves cannot be directly and strictly compared, it is necessary to convert the evaluation results into comparable evaluation values ​​before comparison. Specifically, the evaluation value of the candidate solution can be determined based on the evaluation results and the current optimal time.

[0080] The current optimal time is the shortest runtime among the known runtimes of some feasible solutions on the target hardware at the current moment. Because steps S104 to S112 in this method are executed in a loop and need to be repeated multiple times, the runtimes of some known feasible solutions on the target hardware may exist during this step. Among these known runtimes, the shortest runtime is taken as the current optimal time.

[0081] Additionally, when determining the evaluation value of a candidate solution based on the evaluation result and the current optimal time, there may be a variety of different methods. This disclosure provides a specific embodiment for reference. Specifically, the evaluation value of a candidate solution may be determined according to the following formula:

[0082] Where EI represents the evaluation value, x represents the candidate solution, u(x) represents the average running time in the evaluation result, σ(x) represents the standard deviation in the evaluation result, and f * is the current optimal time; z=(u(x)-f * ) / σ(x), Φ(z) and Denote the probability distribution function and density function of the standard normal distribution respectively. The above formula can be used to obtain the evaluation value of a candidate solution.

[0083] After the evaluation values ​​of all selected candidate solutions of an operator are determined through the evaluation model, the candidate solution with the highest evaluation value can be selected as the pending solution of the operator to be used in subsequent steps.

[0084] S106: Determine the runtime of the target hardware to run the undetermined solution, and increase the number of iterations.

[0085] In this step, for each operator in the computation graph, the runtime of the proposed solution for that operator on the target hardware is determined, while increasing the number of iterations. Specifically, the application corresponding to an operator under the proposed solution—that is, the code that implements the operator using the proposed solution—is input into the target hardware. The runtime of the proposed solution for that operator on the target hardware is determined as the time required for the target hardware to execute the code.

[0086] In the operator optimization method provided in the present disclosure, the number of iterations is the number of times steps S104 to S112 are executed in a loop. Typically, when executing this method, the number of iterations starts at 0 and increases by 1 in step S106 each time the loop from steps S104 to S112 is executed.

[0087] S108: When the running time is less than the current optimal time or the current optimal time does not exist, the running time is determined as the current optimal time, and the undetermined solution is determined as the current optimal solution.

[0088] In step S106, the runtime of the proposed solution for the operator on the target hardware is determined. The ultimate goal of operator optimization is to find a feasible solution with the shortest possible runtime for the operators in the computation graph. Therefore, different approaches can be used to address the runtime differences of the proposed solution for the operator on the target hardware.

[0089] When the running time of the operator's pending solution in the target hardware is less than the current optimal time, or when there is no current optimal time, the running time of the operator's pending solution in the target hardware can be determined as the current optimal time, and the operator's pending solution can be determined as the current optimal solution of the operator. When the running time is less than the current optimal time, it means that the operator's pending solution can better adapt to the target hardware and is better than the existing current optimal solution. Therefore, the running time of the operator's pending solution can be used to replace the current optimal time as the new current optimal time, and the operator's pending solution can be used to replace the current optimal solution as the new current optimal solution. In the case where there is no current optimal time, it usually only occurs in the first round of loops, and no feasible solution has yet been run in the target hardware. At this time, the running time of the operator's pending solution can be directly used as the current optimal time, and the operator's pending solution can be used as the current optimal solution.

[0090] In another case, when the running time of the operator's pending solution is not less than the current optimal time, it means that the operator's pending solution is not better than the current optimal solution. Therefore, the operator's pending solution can be directly discarded without any processing and the subsequent steps can be directly executed.

[0091] S110: When the number of iterations is less than a specified number, reselect a specified number of unselected feasible solutions in the search space of the operator as candidate solutions.

[0092] To ensure the execution efficiency of the operator optimization method provided by the present disclosure, when executing the loop iteration of steps S104 to S112, the number of iterations is capped, that is, the number of iterations will not exceed a specified number. The specified number can be set according to specific needs.

[0093] When the number of iterations has not reached the upper limit, that is, the number of iterations is less than the specified number, return to step S104, reselect a specified number of unselected feasible solutions in the search space of the operator as candidate solutions, and re-execute steps S104 to S112.

[0094] S112: When the number of iterations is not less than the specified number, the current optimal solution is determined as the optimal solution of the operator.

[0095] For each operator, when the number of iterations reaches the upper limit (that is, when the number of iterations is not less than the specified number), the current optimal solution, ultimately determined after multiple iterations, can be determined as the optimal solution for that operator. At this point, this method completes the optimization of the operators in the computational graph. In application, the optimal solution for the operator determined in this step is directly applied to the target neural network model. This achieves optimal adaptation when implementing the target neural network model on the target hardware, allowing operations in the target neural network model to be performed with high efficiency.

[0096] When optimizing operators in the computational graph of a target neural network model using the operator optimization method provided by this disclosure, a locally optimal solution can be found by iterating through the operator's search space, selecting a specified number of candidate solutions without duplication in each iteration. Ultimately, a globally optimal solution is found among the multiple locally optimal solutions, serving as the optimal solution for the operator. Compared to traditional methods, the operator optimization method provided by this disclosure, based on the concept of Bayesian optimization, searches for potential optimal solutions through pre-evaluation. This method can significantly reduce the number of times different feasible solutions of the operator are run on the target hardware, significantly reducing the time cost and overhead of operator optimization.

[0097] Additionally, the evaluation model used in the present disclosure can be pre-trained. Specifically, a sample operator can be obtained; a sample feasible solution of the sample operator and the actual running time of the sample feasible solution in the target hardware can be determined; for each prediction model, the sample feasible solution is input into the prediction model to obtain the predicted running time to be optimized output by the prediction model, wherein the sample feasible solution input into each prediction model belongs to a different sample operator; and the prediction model is trained with the optimization goal of minimizing the difference between the sample predicted running time to be optimized and the actual running time of the sample feasible solution input into the prediction model in the target hardware.

[0098] When the evaluation model is based on a random forest model, training the evaluation model effectively translates into training the individual prediction models included in the evaluation model. The multiple prediction models within the evaluation model cannot be identical; they must have different parameters so that each prediction model can produce distinct outputs for the same input. Therefore, while each prediction model can be trained using the same method, it must be trained using different sample operators. This allows each trained prediction model to produce different outputs for the same input. Of course, in an ideal world, even prediction models trained using different operator samples would likely share the same parameters. However, this ideal world is unlikely to exist in real applications. Therefore, the outputs of each prediction model for the same input are likely to be similar, but rarely identical.

[0099] To further improve the effectiveness of operator optimization, the undetermined solution determined in step S104 can be used as a sample feasible solution, and the evaluation model can be further trained based on the runtime of the undetermined solution on the target hardware determined in step S106. In other words, during actual application, the evaluation model can be continuously optimized and adjusted to achieve the best possible operator optimization results.

[0100] The above is the operator optimization method provided by the present disclosure. Based on the same idea, the embodiment of the present disclosure also provides a corresponding operator optimization device. Figure 2 is a schematic diagram of an operator optimization device provided by an embodiment of the present disclosure. As shown in Figure 2, the operator optimization device includes: an acquisition module 200, a determination module 202, a selection module 204, an addition module 206, an update module 208, an iteration module 210 and an optimization module 212.

[0101] The acquisition module 200 is used to acquire a target neural network model and determine a computational graph of the target neural network model.

[0102] The determination module 202 is configured to determine, for each operator in the computation graph, a search space of the operator, where the search space includes all feasible solutions of the operator.

[0103] The selection module 204 is used to select a specified number of feasible solutions in the search space of the operator as candidate solutions, determine the evaluation value of each candidate solution, and select the candidate solution with the highest evaluation value as the pending solution.

[0104] The increasing module 206 is used to determine the running time of the target hardware to run the undetermined solution and increase the number of iterations.

[0105] The updating module 208 is configured to determine the running time as the current optimal time and the pending solution as the current optimal solution when the running time is less than the current optimal time or the current optimal time does not exist.

[0106] The iteration module 210 is configured to reselect a specified number of unselected candidate solutions in the search space of the operator when the number of iterations is less than a specified number.

[0107] The optimization module 212 is configured to determine the current optimal solution as the optimal solution of the operator when the number of iterations is not less than the specified number.

[0108] Optionally, the selection module 204 is specifically used to select a specified solution from the feasible solutions of the operator as a candidate solution; determine the neighborhood solutions of the specified solution from the feasible solutions of the operator, and re-determine the neighborhood solutions as the specified solutions until the number of determined candidate solutions reaches a specified number.

[0109] Optionally, the selection module 204 is specifically configured to input each candidate solution into a pre-trained evaluation model to obtain an evaluation result of the candidate solution output by the evaluation model; and determine an evaluation value of the candidate solution based on the evaluation result.

[0110] Optionally, the evaluation model includes several prediction models; the selection module 204 is specifically used to input the candidate solution into each prediction model in the pre-trained evaluation model to obtain the predicted running time output by each prediction model; and determine the evaluation result of the candidate solution according to each predicted running time.

[0111] Optionally, the evaluation result includes an average running time and a standard deviation; the selection module 204 is specifically configured to determine the average running time and the standard deviation of the candidate solution according to each predicted running time.

[0112] Optionally, the selection module 204 is specifically configured to determine an evaluation value of the candidate solution based on the evaluation result and the current optimal time.

[0113] Optionally, the device also includes a training module 214, which is specifically used to obtain a sample operator; determine a sample feasible solution of the sample operator, and the actual running time of the sample feasible solution in the target hardware; for each prediction model, input the sample feasible solution into the prediction model to obtain the predicted running time to be optimized output by the prediction model, wherein the sample feasible solution input into each prediction model belongs to a different sample operator; and train the prediction model with the optimization goal of minimizing the difference between the sample predicted running time to be optimized and the actual running time of the sample feasible solution input into the prediction model in the target hardware.

[0114] The present disclosure also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the operator optimization method provided in FIG. 1 above.

[0115] The present disclosure also provides a schematic structural diagram of an electronic device shown in FIG3 . As shown in FIG3 , at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the operator optimization method described in FIG1 above. Of course, in addition to software implementation, the present disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0116] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system onto a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0117] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0118] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0120] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0122] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0124] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0126] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0128] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0130] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0131] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. An operator optimization method, comprising: Obtaining a target neural network model, and determining a computational graph of the target neural network model; For each operator in the computation graph, determining a search space of the operator, wherein the search space contains all feasible solutions of the operator; Selecting a specified number of feasible solutions as candidate solutions in the search space of the operator, determining an evaluation value of each candidate solution, and taking the candidate solution with the highest evaluation value as a pending solution; Determine the running time of the target hardware to run the undetermined solution, and increase the number of iterations; When the running time is less than the current optimal time or there is no current optimal time, the running time is determined as the current optimal time, and the undetermined solution is determined as the current optimal solution; When the number of iterations is less than the specified number, reselecting a specified number of candidate solutions that have not been selected in the search space of the operator; When the number of iterations is not less than the specified number, the current optimal solution is determined as the optimal solution of the operator.

2. The method according to claim 1, characterized in that Selects a specified number of candidate solutions in the search space of this operator, including: Select a specified solution from the feasible solutions of the operator as a candidate solution; A neighborhood solution of the designated solution is determined in the feasible solutions of the operator, and the neighborhood solution is re-determined as the designated solution until the number of determined candidate solutions reaches a designated number.

3. The method according to claim 1, characterized in that Determine the evaluation value of each candidate solution, including: For each candidate solution, the candidate solution is input into a pre-trained evaluation model to obtain an evaluation result of the candidate solution output by the evaluation model; According to the evaluation result, an evaluation value of the candidate solution is determined.

4. The method according to claim 3, characterized in that The evaluation model includes several prediction models; The candidate solution is input into a pre-trained evaluation model to obtain an evaluation result of the candidate solution output by the evaluation model, including: The candidate solution is input into each prediction model in the pre-trained evaluation model to obtain the predicted running time output by each prediction model; The evaluation result of the candidate solution is determined according to each prediction running time.

5. The method according to claim 4, characterized in that The evaluation results include average running time and standard deviation; The evaluation result of the candidate solution is determined according to each prediction running time, including: The average running time and standard deviation of the candidate solution are determined based on the predicted running times.

6. The method according to claim 3, characterized in that Determining an evaluation value of the candidate solution according to the evaluation result includes: An evaluation value of the candidate solution is determined according to the evaluation result and the current optimal time.

7. The method according to claim 4, characterized in that Pre-trained evaluation models, including: Get sample operator; Determining a sample feasible solution of the sample operator and a real running time of the sample feasible solution in target hardware; For each prediction model, the sample feasible solution is input into the prediction model to obtain the prediction running time to be optimized output by the prediction model, wherein the sample feasible solution input into each prediction model belongs to different sample operators; The prediction model is trained with the optimization goal of minimizing the difference between the predicted running time of the sample to be optimized and the actual running time of the sample feasible solution input into the prediction model in the target hardware.

8. An operator optimization device, comprising: An acquisition module, used to acquire a target neural network model and determine a computational graph of the target neural network model; A determination module, configured to determine, for each operator in the computation graph, a search space of the operator, wherein the search space contains all feasible solutions of the operator; A selection module is used to select a specified number of feasible solutions as candidate solutions in the search space of the operator, determine the evaluation value of each candidate solution, and select the candidate solution with the highest evaluation value as the pending solution; Adding a module for determining the running time of the target hardware to run the undetermined solution and increasing the number of iterations; An updating module, configured to determine the running time as the current optimal time and the pending solution as the current optimal solution when the running time is less than the current optimal time or the current optimal time does not exist; An iteration module, configured to reselect a specified number of candidate solutions that have not been selected in the search space of the operator when the number of iterations is less than a specified number; The optimization module is used to determine the current optimal solution as the optimal solution of the operator when the number of iterations is not less than the specified number.

9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images