Operator optimization method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides an operator optimization method, device, electronic device and storage medium, which relates to the field of computer technology. The method includes: in the process of deep learning model reasoning, obtaining the memory usage and calculation time of the calculation process of the operator to be optimized; when the memory usage exceeds the on-chip memory, based on the memory usage and calculation time of the operator to be optimized, optimizing the operator to be optimized in an optimization method that minimizes the running time, wherein the running time includes the calculation time and data interaction time, and the data interaction time is positively correlated with the memory usage. The embodiment of the present application realizes the improvement of the reasoning speed of the deep learning model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an operator optimization method, device, electronic device and storage medium. Background Art

[0002] The memory wall bottleneck refers to the fact that AI hardware assigns computing functions and storage functions to the CPU and memory respectively. The CPU performance has been improving year by year, but the memory performance has not kept up, resulting in a growing performance gap between the two, limiting the improvement and optimization of computing power. In the application scenario of deep learning model inference, operators are usually fused and optimized based on on-chip memory in the hope of improving the inference speed of deep learning models.

[0003] When it comes to operator optimization goals, low computational time is usually used as a single indicator. However, in actual applications, due to limitations in memory and computing power, operators with low computational time do not necessarily improve the inference speed of deep learning models. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an operator optimization method, device, electronic device and storage medium to improve the reasoning speed of the deep learning model. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides an operator optimization method, including:

[0006] During the deep learning model inference process, obtain the memory usage and computing time of the operator to be optimized;

[0007] In the case that the memory usage exceeds the on-chip memory, the operator to be optimized is optimized in an optimization method that minimizes the running time based on the memory usage and the calculation time of the operator to be optimized, wherein the running time includes the calculation time and the data interaction time, and the data interaction time is positively correlated with the memory usage.

[0008] In one embodiment of the present application, when the memory usage exceeds the on-chip memory, the operator to be optimized is optimized in an optimization manner that minimizes the running time based on the memory usage and the calculation time of the operator to be optimized, including:

[0009] In the case where the memory usage exceeds the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension;

[0010] Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and obtaining the data interaction time consumption of each scheduling strategy configuration;

[0011] Using the trained cost model to predict the computational time of each scheduling strategy configuration;

[0012] Comparing the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time;

[0013] The operator to be optimized is optimized according to the target scheduling strategy configuration.

[0014] In one embodiment of the present application, the optimal solution for minimizing the running time is that the data interaction time is minimized and the calculation time is minimized, and the suboptimal solution for minimizing the running time is that the data interaction time is minimized and the calculation time is not minimized.

[0015] In one embodiment of the present application, the comparison of the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time includes:

[0016] For each of the scheduling policy configurations, the data interaction time and the calculation time of the scheduling policy configuration are compared with the current minimized running time. When the data interaction time and the calculation time of the scheduling policy configuration are reduced compared with the current minimized running time, the scheduling policy configuration is updated to a more optimal solution for the current minimized running time; when the data interaction time of the scheduling policy configuration is reduced and the calculation time is increased compared with the current minimized running time, the scheduling policy configuration is updated to a suboptimal solution for the current minimized running time.

[0017] In one embodiment of the present application, after acquiring the basic information of the operator to be optimized and generating the scheduling search space of the operator to be optimized according to a preset scheduling template dimension based on the basic information, the method further includes:

[0018] Disabling the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0019] According to the basic information of the operator to be optimized and the on-chip memory, the memory occupancy is data segmented in a manner that maximizes the reading of data from the external storage to the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width.

[0020] In one embodiment of the present application, the method further includes:

[0021] In a case where the memory usage does not exceed the on-chip memory, the operator to be optimized is optimized in an optimization manner that minimizes the computational time based on the computational time of the operator to be optimized.

[0022] In one embodiment of the present application, when the memory usage does not exceed the on-chip memory, the operator to be optimized is optimized in an optimization manner that minimizes the computational time based on the computational time of the operator to be optimized, including:

[0023] When the memory usage does not exceed the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension;

[0024] Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and predicting the computational time of each scheduling strategy configuration using a trained cost model;

[0025] Comparing the computation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the computation time;

[0026] The operator to be optimized is optimized according to the target scheduling strategy configuration.

[0027] In one embodiment of the present application, after acquiring the basic information of the operator to be optimized and generating the scheduling search space of the operator to be optimized according to a preset scheduling template dimension based on the basic information, the method further includes:

[0028] Enabling a double buffer mechanism of the on-chip memory and a double buffer mechanism between the external storage and the on-chip memory;

[0029] According to the basic information of the operator to be optimized and the on-chip memory, the memory occupancy is divided into data in a manner that maximizes the amount of data calculated at a single time by the hardware unit of the on-chip memory, wherein the direction of the data division includes the number of data blocks after division, the number of channels, the height and the width.

[0030] In one embodiment of the present application, the cost model is trained as follows:

[0031] Obtain the initialized cost model and the sample basic information of the sample operator;

[0032] Based on the sample basic information, generating a scheduling search space of the sample operator according to a preset scheduling template dimension;

[0033] Searching for multiple scheduling strategy configurations of the sample operator in the scheduling search space, and sampling actual time consumption of each of the scheduling strategy configurations;

[0034] Predicting the predicted time consumption of each of the scheduling strategy configurations using the initialized cost model;

[0035] The cost model is trained based on the difference between the predicted time consumption and the actual time consumption.

[0036] In a second aspect, the embodiment of the present application further provides an operator optimization device, including:

[0037] The data acquisition module is used to obtain the memory usage and computing time of the operator to be optimized during the deep learning model inference process;

[0038] The first operator optimization module is used to optimize the operator to be optimized in an optimization method that minimizes the running time based on the memory usage and calculation time of the operator to be optimized when the memory usage exceeds the on-chip memory, wherein the running time includes the calculation time and data interaction time, and the data interaction time is positively correlated with the memory usage.

[0039] In one embodiment of the present application, the first operator optimization module includes:

[0040] A scheduling search space generation submodule, used for obtaining basic information of the operator to be optimized when the memory usage exceeds the on-chip memory, and generating a scheduling search space of the operator to be optimized based on the basic information and according to a preset scheduling template dimension;

[0041] A data interaction time acquisition submodule is used to search for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space to obtain the data interaction time of each scheduling strategy configuration;

[0042] A computation time acquisition submodule is used to predict the computation time of each scheduling strategy configuration using a trained cost model;

[0043] A target scheduling strategy configuration determination submodule is used to compare the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time;

[0044] The operator optimization submodule is used to optimize the operator to be optimized according to the target scheduling strategy configuration.

[0045] In one embodiment of the present application, the target scheduling strategy configuration determination submodule is specifically used to:

[0046] For each of the scheduling policy configurations, the data interaction time and the calculation time of the scheduling policy configuration are compared with the current minimized running time. When the data interaction time and the calculation time of the scheduling policy configuration are reduced compared with the current minimized running time, the scheduling policy configuration is updated to a more optimal solution for the current minimized running time; when the data interaction time of the scheduling policy configuration is reduced and the calculation time is increased compared with the current minimized running time, the scheduling policy configuration is updated to a suboptimal solution for the current minimized running time.

[0047] In one embodiment of the present application, the device further includes:

[0048] A first mechanism switch module, used to turn off the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0049] The first data segmentation module is used to perform data segmentation on the memory occupancy according to the basic information of the operator to be optimized and the on-chip memory in a manner of maximizing the reading of data stored externally to the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width.

[0050] In one embodiment of the present application, the device further includes:

[0051] The second operator optimization module is used to optimize the operator to be optimized in an optimization manner that minimizes the calculation time based on the calculation time of the operator to be optimized when the memory occupancy does not exceed the on-chip memory.

[0052] In one embodiment of the present application, the second operator optimization module is specifically used to:

[0053] When the memory usage does not exceed the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension;

[0054] Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and predicting the computational time of each scheduling strategy configuration using a trained cost model;

[0055] Comparing the computation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the computation time;

[0056] The operator to be optimized is optimized according to the target scheduling strategy configuration.

[0057] In one embodiment of the present application, the device further includes:

[0058] A second mechanism switch module, used to enable the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0059] The second data segmentation module is used to perform data segmentation on the memory occupancy according to the basic information of the operator to be optimized and the on-chip memory in a manner that maximizes the amount of data calculated at a single time by the hardware unit of the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width.

[0060] In one embodiment of the present application, the apparatus further includes: a cost model training module, configured to:

[0061] Obtain the initialized cost model and the sample basic information of the sample operator;

[0062] Based on the sample basic information, generating a scheduling search space of the sample operator according to a preset scheduling template dimension;

[0063] Searching for multiple scheduling strategy configurations of the sample operator in the scheduling search space, and sampling actual time consumption of each of the scheduling strategy configurations;

[0064] Predicting the predicted time consumption of each of the scheduling strategy configurations using the initialized cost model;

[0065] The cost model is trained based on the difference between the predicted time consumption and the actual time consumption.

[0066] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0067] Memory, used to store computer programs;

[0068] The processor is used to implement any of the above-mentioned operator optimization methods when executing the program stored in the memory.

[0069] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned operator optimization methods.

[0070] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned operator optimization methods.

[0071] Beneficial effects of the embodiments of the present application:

[0072] The operator optimization method provided in the embodiment of the present application is that when the memory usage exceeds the on-chip memory, the calculation of the operator to be optimized will overflow to the external memory. Based on the memory usage and calculation time of the operator to be optimized, the operator to be optimized is optimized in an optimization method that minimizes the running time. The running time includes the calculation time and the data interaction time. The data interaction time is positively correlated with the memory usage. The more interactive IO data, the more bandwidth is occupied, and the more data interaction time is consumed. Minimizing the running time can minimize the sum of the data interaction time and the calculation time, and at the same time optimizes the operator's time and bandwidth, reduces the probability of deep learning model reasoning occupying the external memory bandwidth at a high frequency due to excessive bandwidth occupation, and expands the applicable scenarios of the optimized operator, especially for bandwidth bottleneck scenarios.

[0073] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0075] Figure 1-1 A schematic diagram of the process flow of the first operator optimization method provided in the embodiment of the present application;

[0076] Figure 1-2 An example diagram of a fusion operator development provided in an embodiment of the present application;

[0077] Figure 2 A possible implementation of step S102 provided in an embodiment of the present application;

[0078] Figure 3-1 A possible implementation of step S201 provided in an embodiment of the present application;

[0079] Figure 3-2 An example diagram of a bandwidth-first optimization strategy application provided in an embodiment of the present application;

[0080] Figure 4-1 A schematic diagram of a flow chart of a second operator optimization method provided in an embodiment of the present application;

[0081] Figure 4-2 An example diagram of an application of an optimization strategy selection provided in an embodiment of the present application;

[0082] Figure 5 A possible implementation of step S103 provided in an embodiment of the present application;

[0083] Figure 6-1 A possible implementation of step S401 provided in an embodiment of the present application;

[0084] Figure 6-2 An example diagram of an optimization strategy application with time-consuming priority provided in an embodiment of the present application;

[0085] Figure 7-1 A flowchart of a cost model training method provided in an embodiment of the present application;

[0086] Figure 7-2 An example diagram of a training application of a cost model provided in an embodiment of the present application;

[0087] Figure 7-3 An example diagram of an application of an operator optimization method provided in an embodiment of the present application;

[0088] Figure 8 A schematic diagram of the structure of an operator optimization device provided in an embodiment of the present application;

[0089] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0091] The following are explanations of professional terms that may appear in this application:

[0092] Deep learning: Deep learning is a branch of machine learning. It is an algorithm that attempts to use multiple processing layers with complex structures or multiple nonlinear transformations to perform high-level abstraction on data. Deep learning is an algorithm in machine learning that is based on learning to represent data. So far, there are several deep learning frameworks, such as convolutional neural networks, deep belief networks, and recursive neural networks, which have been applied in computer vision, speech recognition, natural language processing, audio recognition, and bioinformatics, and have achieved excellent results.

[0093] Fusion operator: The deep learning network model is essentially a directed graph composed of operators of different types, shapes, and connection relationships; the fusion operator moves the data transmission and calculation process between operators to the on-chip memory of the AI ​​hardware.

[0094] Combination of hardware characteristics: During the network model optimization process, performance optimization is performed by perceiving the characteristics of AI hardware, such as computing unit accuracy, computing unit computing power, computing unit data arrangement format, on-chip memory size, external memory bandwidth, external memory IO data volume, etc.

[0095] Double buffer mechanism: The double buffer mechanism of on-chip memory refers to the double buffer setting of data sender and data receiver between operators; the double buffer mechanism between on-chip memory and external storage refers to the double buffer setting of on-chip memory and external storage for reading data.

[0096] On-chip memory refers to the memory in the system integrated circuit that AI hardware allocates to deep learning models for reasoning. In addition to on-chip memory, AI hardware also has external memory for processing other intelligent applications installed on AI hardware. The reasoning of deep learning models is performed on on-chip memory, but when the memory required for reasoning exceeds the on-chip memory, the memory of the external memory may be preempted to implement the reasoning of the deep learning model, resulting in processing delays in the overall operation of the AI ​​hardware. Especially in scenarios where the memory bandwidth is limited, such as the CV Transformer (a neural network model based on the self-attention mechanism in the field of computer vision) network model, the end-side SoC (system integrated chip, Application-Specific SoC) intelligent hardware, etc., the bandwidth occupied by the network model is likely to become a bottleneck, especially in concurrent intelligent application scenarios, where the concurrent processing of different hardware units and the concurrency of multiple intelligent applications will frequently preempt the bandwidth of the external memory.

[0097] In order to solve this problem, the embodiments of the present application provide an operator optimization method, device, electronic device and storage medium.

[0098] The following is a detailed description of the operator optimization method provided in the embodiment of the present application. Figure 1-1 , Figure 1-1 A flowchart of a first operator optimization method is provided for an embodiment of the present application, including:

[0099] Step S101, during the deep learning model inference process, obtain the memory usage and computing time of the operator calculation process to be optimized.

[0100] The operator to be optimized refers to the fusion operator in the deep learning model, which includes multiple basic operators, and the basic operators are connected with each other. Each basic operator has different or same type and shape.

[0101] In an example, the fusion operator can be developed according to actual needs and implemented by the fusion operator development module, which consists of the architecture layer, basic layer, fusion layer, segmentation layer, and operator layer. Figure 1-2 shown.

[0102] Among them, the architecture layer is used to encapsulate the hardware bottom-level instructions, which are divided into four types: Martirx (image transformation), Vector (vector), Scalar (scalar), and Memory (storage), and perform basic on-chip operator encapsulation to shield the differences in the bottom-level instructions of each hardware. The Matrix module is used to encapsulate basic convolution and matrix multiplication calculation instructions; the Vector module is used to encapsulate vector calculation operation instructions, such as Add (addition), Mul (multiplication), Sub (subtraction), Max (maximum value), Min (minimum value), And (and operation), Or (or operation), etc.; the Scalar module is used to encapsulate scalar calculation operation instructions, such as logic control operations, single data operations of addition, subtraction, multiplication and division, etc.; the Memory module is used to encapsulate on-chip memory operation instructions, including asynchronous copy, synchronous copy, L1 / L2 / L3 / shared memory and other on-chip memory operations, on-chip memory usage statistics, etc.

[0103] The base layer is used to implement the basic on-chip operators of conventional network models according to the architecture layer encapsulation, such as the conv (convolution), pooling (pooling), relu (corrected linear), softmax (normalization) and other network layers of the convolutional neural network; and the Matmul (matrix multiplication), layerNormal (layer normalization), Gelu (activation), Transpose (transposition) and other network layers of the CVTransformer network; it is implemented by hand-written templated on-chip operators, such as 3 3.1 1 specific convolution operator, 32 32, 32 64, 64 64 equal-size blocks of matrix multiplication operators, and generic implementation operators that do not distinguish between specific sizes.

[0104] The fusion layer is encapsulated based on the base layer to implement templated fusion on-chip operators for conventional network models. The fusion operators include basic operator operations such as Matrix, Vector, Scalar, and Memory (Matrix+Vector+Memory+Scalar); for example, convolutional neural network fusion operators such as conv+conv, conv+pooling, conv+pooling+relu, and CVTransformer network fusion operators such as Matmul+Matmul, layernormal+Matmul, layernormal+Matmul+Gelu+Transpose.

[0105] The data interaction between fusion operators built based on the fusion layer is completed in the on-chip memory, not through DDR (external memory).

[0106] The slicing layer is based on the fusion layer encapsulation and provides a templated data slicing mechanism between the on-chip fusion operator and the external storage, including external storage data slicing: the tiling (data block) size of a single data read; data slicing direction, including N direction (number), C direction (number of channels), H direction (height), W direction (width) and other slicing methods; a double buffer (buffer, background buffer and foreground buffer) buffering mechanism between the external storage and the on-chip operator to implement ping-pong operation; a data block alignment mechanism combined with hardware computing characteristics, such as 16, 32, 64, 128 size alignment, etc.

[0107] The operator layer is encapsulated based on the slice layer, which is the kernel operator that can be actually executed by the hardware. Based on this layer, the fused operator is called to realize the data calculation function. For example, the convolutional neural network conv+conv, conv+pooling, conv+pooling+relu, and Attention, ffn and other fusion operators, as well as the CV Transformer network Matmul+Matmul, layernormal+Matmul, layernormal+Matmul+Gelu+Transpose and other fusion operators.

[0108] It provides five levels of fusion operator development module solutions, uses handwritten optimized on-chip operators as basic ops to quickly build various complex fusion operators, abstracts double buffering, data tiling block segmentation, data segmentation direction, etc. into template solutions, and supports opening / closing and selection.

[0109] During the inference process of the deep learning model on AI hardware, the computation time and memory usage of the operator to be optimized are calculated based on the type, shape and connection relationship of each basic operator. The computation time refers to the total time required for the operator to be optimized to run during the inference process of the deep learning model; the memory usage refers to the memory required for the data IO (in / out, input and output) interaction between the basic operators during the inference process of the deep learning model, that is, the amount of interactive IO data.

[0110] Step S102, when the memory usage exceeds the on-chip memory, the operator to be optimized is optimized in an optimization method that minimizes the running time based on the memory usage and calculation time of the operator to be optimized, wherein the running time includes the calculation time and data interaction time, and the data interaction time is positively correlated with the memory usage.

[0111] When the memory usage of the operator to be optimized exceeds the on-chip memory, it means that the deep learning model reasoning not only needs to occupy the on-chip memory, but also overflows to the external memory. At this time, based on the memory usage and computing time, the optimization operator to be optimized is optimized in an optimization method that minimizes the running time. Specifically, the running time includes computing time and data interaction time. The positive correlation between data interaction time and memory usage means that the more interactive IO data, the more bandwidth is occupied and the more data interaction time is consumed. The optimization method that minimizes the running time means giving priority to bandwidth.

[0112] In one embodiment of the present application, the optimal solution for minimizing the running time is to minimize the data interaction time and the calculation time, and the suboptimal solution for minimizing the running time is to minimize the data interaction time and the calculation time is not minimized.

[0113] When both the data interaction time and the computation time are minimized, the running time is minimized, which is the optimal solution. When the data interaction time is minimized, but the computation time is not minimized, it can be used as a suboptimal solution because it is a bandwidth-first optimization method. That is, when the operator cannot be optimized according to the optimal solution, the operator is optimized according to the suboptimal solution.

[0114] It can be seen from the above that the operator optimization method provided by the embodiment of the present application, when the memory usage exceeds the on-chip memory, the calculation of the operator to be optimized will overflow to the external memory, and the operator to be optimized is optimized in an optimization manner that minimizes the running time based on the memory usage and calculation time of the operator to be optimized. The running time includes calculation time and data interaction time, and the data interaction time is positively correlated with the memory usage. The more interactive IO data, the more bandwidth is occupied, and the more data interaction time is consumed. Minimizing the running time can minimize the sum of data interaction time and calculation time, while optimizing the operator's time and bandwidth, improving the speed of deep learning model reasoning, reducing the probability of deep learning model reasoning occupying too much bandwidth and high-frequency external memory bandwidth, and expanding the applicable scenarios of the optimized operator, especially for bandwidth bottleneck scenarios.

[0115] In one embodiment of the present application, Figure 2 As shown, in the above step S102, when the memory usage exceeds the on-chip memory, the operator to be optimized is optimized according to the optimization method of minimizing the running time based on the memory usage and calculation time of the operator to be optimized, including:

[0116] Step S201, when the memory usage exceeds the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension;

[0117] Step S202, searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and obtaining the data interaction time consumption of each scheduling strategy configuration;

[0118] Step S203, using the trained cost model to predict the computation time of each scheduling strategy configuration;

[0119] Step S204, comparing the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time;

[0120] Step S205, optimizing the operator to be optimized according to the target scheduling strategy configuration.

[0121] The preset scheduling template dimensions are shown in Table 1. The scheduling strategy configuration refers to configuring each item in the scheduling template dimension according to different parameters, such as turning on or off the on-chip double buffer mechanism switch, different values ​​of data block size, different data segmentation directions, etc.

[0122] Table 1 Scheduling template dimensions

[0123]

[0124] The basic information of the operator to be optimized includes shape, type, network layer parameters, connection relationship between basic operators, etc. Based on this basic information, the scheduling search space of the operator to be optimized is established according to the scheduling template dimension. Specifically, the scheduling search space can be configured according to actual needs, such as setting the parameter value range, positive correlation or negative correlation between parameters, etc.

[0125] Search for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space. The parameters in each dimension corresponding to each configuration are different. The corresponding interactive IO data volume is obtained according to each scheduling strategy configuration to determine the data interaction time.

[0126] The trained cost model is used to predict the computational time of each scheduling strategy configuration. The data interaction time and computational time of each scheduling strategy configuration are compared to obtain the target scheduling strategy configuration that minimizes the running time.

[0127] In one embodiment of the present application, the above step S204 compares the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time, including:

[0128] For each of the scheduling policy configurations, the data interaction time and the calculation time of the scheduling policy configuration are compared with the current minimized running time. When the data interaction time and the calculation time of the scheduling policy configuration are reduced compared with the current minimized running time, the scheduling policy configuration is updated to a more optimal solution for the current minimized running time; when the data interaction time of the scheduling policy configuration is reduced and the calculation time is increased compared with the current minimized running time, the scheduling policy configuration is updated to a suboptimal solution for the current minimized running time.

[0129] Specifically, the scheduling strategy obtained by the first search in the scheduling search space is configured as the initial scheduling strategy, the first data interaction time of the initial scheduling strategy is determined, and the first calculation time of the initial scheduling strategy is predicted using the trained cost model as the initial value of the optimal solution and the suboptimal solution.

[0130] Continue to search for the scheduling policy configuration of the operator to be optimized in the scheduling search space, and compare the computation time and data interaction time corresponding to each scheduling policy configuration searched with the first computation time and the first data interaction time, respectively. If the data interaction time and computation time of the new scheduling policy configuration decrease compared to the initial scheduling policy, it will be updated to a better solution. If the data interaction time of the new scheduling policy configuration decreases and the computation time increases compared to the initial scheduling policy, it will be updated to a suboptimal solution. The others are non-compliant configurations, as shown in Table 2. Each scheduling policy configuration searched is compared with the current better solution and suboptimal solution until the preset search end condition is met, that is, the search number reaches the preset maximum iterative search number or the search of the entire scheduling search space has been completed, then the search ends and the final target scheduling policy configuration (better solution) is obtained. Configure the operator to be optimized according to the target scheduling policy configuration to complete the bandwidth-first optimization of the operator to be optimized.

[0131] Table 2

[0132]

[0133] As can be seen from the above, the operator optimization method provided in the embodiment of the present application establishes a scheduling search space for the operator to be optimized based on a preset scheduling template dimension, thereby accelerating the generation of the scheduling search space; searching for multiple scheduling strategy configurations in the scheduling search space, and ultimately selecting the optimal solution with the smallest data interaction time and calculation time as the target scheduling strategy configuration to optimize the operator, thereby achieving simultaneous consideration of bandwidth and time consumption, and expanding the application scenarios of operator optimization.

[0134] In one embodiment of the present application, Figure 3-1 As shown, after obtaining the basic information of the operator to be optimized in the above step S201, and generating the scheduling search space of the operator to be optimized based on the basic information according to the preset scheduling template dimension, the method further includes:

[0135] Step S301, shutting down the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0136] Step S302 , according to the basic information of the operator to be optimized and the on-chip memory, data segmentation is performed on the memory usage in a manner of maximizing the reading of data stored externally in the on-chip memory.

[0137] The direction of data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width (N, C, H, W).

[0138] The double buffer mechanism of on-chip memory refers to the double buffer setting of data sender and data receiver between operators; the double buffer mechanism between on-chip memory and external storage refers to the double buffer setting of on-chip memory and external storage reading data.

[0139] Data segmentation refers to dividing the IO interactive data into data blocks when reading / writing. The N, C, H, and W of the data block are determined in a way that as much external storage data as possible is read into the on-chip memory.

[0140] like Figure 3-2 As shown, an example diagram of an application of a bandwidth-first optimization strategy is provided.

[0141] As can be seen from the above, the operator optimization method provided in the embodiment of the present application turns off the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory, so as to maximize the use of the on-chip memory to store the intermediate calculation results of the operator to be optimized, and divide the data in the direction of reading as much external storage data as possible to the on-chip memory, thereby reducing the scheduling search space by combining hardware characteristics and bandwidth priority strategy, reducing the size of the scheduling search space, and accelerating the search process of scheduling policy configuration.

[0142] In one embodiment of the present application, Figure 4-1 As shown, the method also includes:

[0143] Step S103 , when the memory usage does not exceed the on-chip memory, based on the computational time of the operator to be optimized, the operator to be optimized is optimized in an optimization manner that minimizes the computational time.

[0144] like Figure 4-2 As shown, the optimization strategy can be selected based on actual needs, i.e., time-consuming priority or bandwidth-consuming priority. When bandwidth-consuming priority is selected, it is determined whether the memory usage does not exceed the on-chip memory, i.e., whether all operators to be optimized can be calculated in the on-chip memory. If so, the optimization is performed based on time-consuming priority.

[0145] In one embodiment of the present application, Figure 5 As shown, in the above step S103, when the memory usage does not exceed the on-chip memory, based on the calculation time of the operator to be optimized, the operator to be optimized is optimized in an optimization method that minimizes the calculation time, including:

[0146] Step S401, when the memory usage does not exceed the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension;

[0147] Step S402, searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and using the trained cost model to predict the computation time of each scheduling strategy;

[0148] Step S403, comparing the computation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the computation time;

[0149] Step S404: optimizing the operator to be optimized according to the target scheduling strategy configuration.

[0150] When the memory usage does not exceed the on-chip memory, it means that the on-chip memory can realize the calculation of the operator to be optimized without overflowing to the external memory, and there is no need to consider the bandwidth usage of the external memory.

[0151] In the scheduling search space, multiple scheduling strategy configurations of the operator to be optimized are searched. The parameters in each dimension corresponding to each configuration are different. The trained cost model is used to predict the computational time of each scheduling strategy configuration. The computational time of each scheduling strategy configuration is compared to obtain the target scheduling strategy configuration that minimizes the computational time.

[0152] Specifically, the scheduling strategy obtained by the first search in the scheduling search space is configured as the initial scheduling strategy, and the first calculation time consumption of the initial scheduling strategy is predicted using the trained cost model as the initial value of the optimal solution.

[0153] Continue to search for the scheduling policy configuration of the operator to be optimized in the scheduling search space, and compare the computational time corresponding to each scheduling policy configuration searched with the first computational time. If the computational time of the new scheduling policy configuration is reduced compared to the initial scheduling policy data, it will be updated to the optimal solution. Each scheduling policy configuration searched is compared with the current optimal solution until the preset search end condition is met, that is, the search number reaches the preset maximum number of iterative searches or the search of the entire scheduling search space has been completed, then the search ends and the final target scheduling policy configuration (optimal solution) is obtained. Configure the operator to be optimized according to the target scheduling policy configuration to complete the bandwidth-first optimization of the operator to be optimized.

[0154] As can be seen from the above, the operator optimization method provided in the embodiment of the present application establishes a scheduling search space for the operator to be optimized based on a preset scheduling template dimension, thereby accelerating the generation of the scheduling search space; searching for multiple scheduling strategy configurations in the scheduling search space, and finally selecting the optimal solution with the least computational time as the target scheduling strategy configuration to optimize the operator, thereby achieving the optimization operator with the lowest time consumption.

[0155] In one embodiment of the present application, Figure 6-1As shown, after obtaining the basic information of the operator to be optimized in step S401, and generating the scheduling search space of the operator to be optimized according to the preset scheduling template dimension based on the basic information, the method further includes:

[0156] Step S501, enabling the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0157] Step S502, according to the basic information of the operator to be optimized and the on-chip memory, the memory occupancy is divided into data in a manner that maximizes the amount of data calculated at a single time by the hardware unit of the on-chip memory, wherein the direction of the data division includes the number of data blocks after division, the number of channels, the height and the width (N, C, H, W).

[0158] The double buffer mechanism of on-chip memory refers to the double buffer setting of data sender and data receiver between operators; the double buffer mechanism between on-chip memory and external storage refers to the double buffer setting of on-chip memory and external storage reading data.

[0159] Data segmentation refers to dividing the IO interactive data into data blocks when reading / writing. The N, C, H, and W of the data block are determined in a way that the amount of data calculated by the hardware unit at a time can be adapted to the computing power of the hardware unit to the greatest extent.

[0160] like Figure 6-2 As shown, an application example diagram of a time-consuming priority optimization strategy is provided.

[0161] As can be seen from the above, the operator optimization method provided in the embodiment of the present application enables the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory, thereby improving the multi-level pipeline parallel capability of the hardware; the data is segmented in such a way that the amount of data calculated by the hardware unit at a single time can be adapted to the computing power of the hardware unit to the greatest extent, thereby reducing the scheduling search space by combining hardware characteristics and bandwidth priority strategy, reducing the size of the scheduling search space, and accelerating the search process of the scheduling policy configuration.

[0162] In one embodiment of the present application, Figure 7-1 As shown, the training method of the cost model is as follows:

[0163] Step S601, obtaining an initialized cost model and obtaining sample basic information of a sample operator;

[0164] Step S602: generating a scheduling search space of the sample operator based on the sample basic information and according to a preset scheduling template dimension;

[0165] Step S603, searching for multiple scheduling strategy configurations of the sample operator in the scheduling search space, and sampling the actual time consumption of each scheduling strategy configuration;

[0166] Step S604, using the initialized cost model to predict the predicted time consumption of each scheduling strategy configuration;

[0167] Step S605: training the cost model based on the difference between the predicted time consumption and the actual time consumption.

[0168] like Figure 7-2 As shown in FIG, a training application example diagram of a cost model is also provided. Figure 7-3 As shown, an example diagram of an operator optimization application is also provided.

[0169] As can be seen from the above, the operator optimization method provided in the embodiment of the present application uses the scheduling template and the computational time as training data to implement the training of the cost model, so as to obtain a cost model that can predict the computational time of the scheduling strategy configuration of the operator to be optimized.

[0170] See also Figure 8 , the embodiment of the present application also provides a structural schematic diagram of an operator optimization device, including:

[0171] The data acquisition module 801 is used to obtain the memory usage and computing time of the operator to be optimized during the deep learning model inference process;

[0172] The first operator optimization module 802 is used to optimize the operator to be optimized in an optimization method that minimizes the running time based on the memory usage and calculation time of the operator to be optimized when the memory usage exceeds the on-chip memory, wherein the running time includes the calculation time and data interaction time, and the data interaction time is positively correlated with the memory usage.

[0173] It can be seen from the above that in the operator optimization device provided by the embodiment of the present application, when the memory usage exceeds the on-chip memory, the calculation of the operator to be optimized will overflow to the external memory. Based on the memory usage and calculation time of the operator to be optimized, the operator to be optimized is optimized in an optimization method that minimizes the running time. The running time includes calculation time and data interaction time, and the data interaction time is positively correlated with the memory usage. The more interactive IO data, the more bandwidth is occupied, and the more data interaction time is consumed. Minimizing the running time can minimize the sum of data interaction time and calculation time, while optimizing the operator's time and bandwidth, improving the speed of deep learning model reasoning, reducing the probability of deep learning model reasoning occupying too much bandwidth and high-frequency external memory bandwidth, and expanding the applicable scenarios of the optimized operator, especially for bandwidth bottleneck scenarios.

[0174] In one embodiment of the present application, the first operator optimization module 802 includes:

[0175] A scheduling search space generation submodule, used for obtaining basic information of the operator to be optimized when the memory usage exceeds the on-chip memory, and generating a scheduling search space of the operator to be optimized based on the basic information and according to a preset scheduling template dimension;

[0176] A data interaction time acquisition submodule is used to search for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space to obtain the data interaction time of each scheduling strategy configuration;

[0177] A computation time acquisition submodule is used to predict the computation time of each scheduling strategy configuration using a trained cost model;

[0178] A target scheduling strategy configuration determination submodule is used to compare the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time;

[0179] The operator optimization submodule is used to optimize the operator to be optimized according to the target scheduling strategy configuration.

[0180] As can be seen from the above, the operator optimization device provided in the embodiment of the present application establishes a scheduling search space for the operator to be optimized based on a preset scheduling template dimension, thereby accelerating the generation of the scheduling search space; searching for multiple scheduling strategy configurations in the scheduling search space, and ultimately selecting the optimal solution with the smallest data interaction time and calculation time as the target scheduling strategy configuration to optimize the operator, thereby achieving simultaneous consideration of bandwidth and time consumption, and expanding the application scenarios of operator optimization.

[0181] In one embodiment of the present application, the target scheduling strategy configuration determination submodule is specifically used to:

[0182] For each of the scheduling policy configurations, the data interaction time and the calculation time of the scheduling policy configuration are compared with the current minimized running time. When the data interaction time and the calculation time of the scheduling policy configuration decrease compared with the current minimized running time, the scheduling policy configuration is updated to a better solution for the current minimized running time; when the data interaction time of the scheduling policy configuration decreases and the calculation time increases compared with the current minimized running time, the scheduling policy configuration is updated to a suboptimal solution for the current minimized running time;

[0183] In one embodiment of the present application, the device further includes:

[0184] A first mechanism switch module, used to turn off the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0185] The first data segmentation module is used to perform data segmentation on the memory occupancy according to the basic information of the operator to be optimized and the on-chip memory in a manner of maximizing the reading of data stored externally to the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width.

[0186] As can be seen from the above, the operator optimization device provided in the embodiment of the present application turns off the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory, so as to maximize the use of the on-chip memory to store the intermediate calculation results of the operator to be optimized, and divide the data in the direction of reading as much external storage data as possible to the on-chip memory, thereby reducing the scheduling search space by combining hardware characteristics and bandwidth priority strategy, reducing the size of the scheduling search space, and accelerating the search process of scheduling policy configuration.

[0187] In one embodiment of the present application, the device further includes:

[0188] The second operator optimization module is used to optimize the operator to be optimized in an optimization manner that minimizes the calculation time based on the calculation time of the operator to be optimized when the memory occupancy does not exceed the on-chip memory.

[0189] In one embodiment of the present application, the second operator optimization module is specifically used to:

[0190] When the memory usage does not exceed the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension;

[0191] Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and predicting the computation time of each scheduling strategy using the trained cost model;

[0192] Comparing the computation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the computation time;

[0193] The operator to be optimized is optimized according to the target scheduling strategy configuration.

[0194] As can be seen from the above, the operator optimization device provided in the embodiment of the present application establishes a scheduling search space for the operator to be optimized based on a preset scheduling template dimension, thereby accelerating the generation of the scheduling search space; searching for multiple scheduling strategy configurations in the scheduling search space, and finally selecting the optimal solution with the least computational time as the target scheduling strategy configuration to optimize the operator, thereby achieving the optimization operator with the lowest time consumption.

[0195] In one embodiment of the present application, the device further includes:

[0196] A second mechanism switch module, used to enable the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory;

[0197] The second data segmentation module is used to perform data segmentation on the memory occupancy according to the basic information of the operator to be optimized and the on-chip memory in a manner that maximizes the amount of data calculated at a single time by the hardware unit of the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width.

[0198] As can be seen from the above, the operator optimization device provided in the embodiment of the present application enables the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory, thereby improving the multi-level pipeline parallel capability of the hardware; data segmentation is implemented in a manner that the amount of data calculated by the hardware unit at a single time can be adapted to the computing power of the hardware unit to the greatest extent, thereby reducing the scheduling search space by combining hardware characteristics and bandwidth priority strategy, reducing the size of the scheduling search space, and accelerating the search process of scheduling policy configuration.

[0199] In one embodiment of the present application, the apparatus further includes: a cost model training module, configured to:

[0200] Obtain the initialized cost model and the sample basic information of the sample operator;

[0201] Based on the sample basic information, generating a scheduling search space of the sample operator according to a preset scheduling template dimension;

[0202] Searching for multiple scheduling strategy configurations of the sample operator in the scheduling search space, and sampling actual time consumption of each of the scheduling strategy configurations;

[0203] Predicting the predicted time consumption of each of the scheduling strategy configurations using the initialized cost model;

[0204] The cost model is trained based on the difference between the predicted time consumption and the actual time consumption.

[0205] As can be seen from the above, the operator optimization device provided in the embodiment of the present application uses the scheduling template and the computational time as training data to implement the training of the cost model, so as to obtain a cost model that can predict the computational time of the scheduling strategy configuration of the operator to be optimized.

[0206] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0207] Memory 901, used for storing computer programs;

[0208] The processor 902 is configured to implement any of the above-mentioned operator optimization methods when executing the program stored in the memory 901.

[0209] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 communicate with each other via the communication bus.

[0210] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0211] The communication interface is used for communication between the above electronic device and other devices.

[0212] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0213] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0214] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned operator optimization methods are implemented.

[0215] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any operator optimization method in the above embodiments.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0217] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0218] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0219] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An operator optimization method, characterized in that: include: During the deep learning model inference process, obtain the memory usage and computing time of the operator to be optimized; In the case where the memory usage exceeds the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension; Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and obtaining data interaction time consumption of each scheduling strategy configuration, wherein the data interaction time consumption is positively correlated with the memory usage; Using the trained cost model to predict the computational time of each scheduling strategy configuration; Comparing the data interaction time and the calculation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the running time, wherein the running time includes the calculation time and the data interaction time; The operator to be optimized is optimized according to the target scheduling strategy configuration.

2. The method according to claim 1, characterized in that The optimal solution for minimizing the running time is that the data interaction time is minimized and the calculation time is minimized, and the suboptimal solution for minimizing the running time is that the data interaction time is minimized and the calculation time is not minimized.

3. The method according to claim 2, characterized in that The comparing the data interaction time and the calculation time of each scheduling strategy configuration to obtain a target scheduling strategy configuration that minimizes the running time includes: For each of the scheduling policy configurations, the data interaction time and the calculation time of the scheduling policy configuration are compared with the current minimized running time. When the data interaction time and the calculation time of the scheduling policy configuration are reduced compared with the current minimized running time, the scheduling policy configuration is updated to a more optimal solution for the current minimized running time; when the data interaction time of the scheduling policy configuration is reduced and the calculation time is increased compared with the current minimized running time, the scheduling policy configuration is updated to a suboptimal solution for the current minimized running time.

4. The method according to claim 1, characterized in that After acquiring the basic information of the operator to be optimized and generating the scheduling search space of the operator to be optimized according to a preset scheduling template dimension based on the basic information, the method further includes: Disabling the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory; According to the basic information of the operator to be optimized and the on-chip memory, the memory occupancy is data segmented in a manner that maximizes the reading of data from the external storage to the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width.

5. The method according to claim 1, characterized in that The method further comprises: In a case where the memory usage does not exceed the on-chip memory, the operator to be optimized is optimized in an optimization manner that minimizes the computational time based on the computational time of the operator to be optimized.

6. The method according to claim 5, characterized in that When the memory usage does not exceed the on-chip memory, the operator to be optimized is optimized in an optimization manner that minimizes the computational time consumption based on the computational time consumption of the operator to be optimized, including: When the memory usage does not exceed the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension; Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and predicting the computational time of each scheduling strategy configuration using a trained cost model; Comparing the computation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the computation time; The operator to be optimized is optimized according to the target scheduling strategy configuration.

7. The method according to claim 6, characterized in that After acquiring the basic information of the operator to be optimized and generating the scheduling search space of the operator to be optimized according to a preset scheduling template dimension based on the basic information, the method further includes: Enabling a double buffer mechanism of the on-chip memory and a double buffer mechanism between the external storage and the on-chip memory; According to the basic information of the operator to be optimized and the on-chip memory, the memory occupancy is divided into data in a manner that maximizes the amount of data calculated at a single time by the hardware unit of the on-chip memory, wherein the direction of the data division includes the number of data blocks after division, the number of channels, the height and the width.

8. The method according to claim 1 or 6, characterized in that: The cost model is trained as follows: Obtain the initialized cost model and the sample basic information of the sample operator; Based on the sample basic information, generating a scheduling search space of the sample operator according to a preset scheduling template dimension; Searching for multiple scheduling strategy configurations of the sample operator in the scheduling search space, and sampling actual time consumption of each of the scheduling strategy configurations; Predicting the predicted time consumption of each of the scheduling strategy configurations using the initialized cost model; The cost model is trained based on the difference between the predicted time consumption and the actual time consumption.

9. An operator optimization device, characterized in that: include: The data acquisition module is used to obtain the memory usage and computing time of the operator to be optimized during the deep learning model inference process; A scheduling search space generation submodule, used for obtaining basic information of the operator to be optimized when the memory usage exceeds the on-chip memory, and generating a scheduling search space of the operator to be optimized based on the basic information and according to a preset scheduling template dimension; A data interaction time acquisition submodule is used to search for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and obtain the data interaction time of each scheduling strategy configuration, wherein the data interaction time is positively correlated with the memory usage; A computation time acquisition submodule is used to predict the computation time of each scheduling strategy configuration using a trained cost model; A target scheduling policy configuration determination submodule is used to compare the data interaction time and the calculation time of each scheduling policy configuration to obtain a target scheduling policy configuration that minimizes the running time, wherein the running time includes the calculation time and the data interaction time; An operator optimization submodule, configured to optimize the operator to be optimized according to the target scheduling strategy; The target scheduling strategy configuration determination submodule is specifically used for: For each of the scheduling policy configurations, the data interaction time and the calculation time of the scheduling policy configuration are compared with the current minimized running time. When the data interaction time and the calculation time of the scheduling policy configuration decrease compared with the current minimized running time, the scheduling policy configuration is updated to a better solution for the current minimized running time; when the data interaction time of the scheduling policy configuration decreases and the calculation time increases compared with the current minimized running time, the scheduling policy configuration is updated to a suboptimal solution for the current minimized running time; The device also includes: A first mechanism switch module, used to turn off the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory; A first data segmentation module is used to perform data segmentation on the memory usage in a manner of maximizing the reading of data stored externally in the on-chip memory according to the basic information of the operator to be optimized and the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width; The device also includes: A second operator optimization module is used to optimize the operator to be optimized in an optimization manner that minimizes the computational time consumption based on the computational time consumption of the operator to be optimized when the memory occupancy does not exceed the on-chip memory; The second operator optimization module is specifically used for: When the memory usage does not exceed the on-chip memory, basic information of the operator to be optimized is obtained, and based on the basic information, a scheduling search space of the operator to be optimized is generated according to a preset scheduling template dimension; Searching for multiple scheduling strategy configurations of the operator to be optimized in the scheduling search space, and predicting the computational time of each scheduling strategy configuration using a trained cost model; Comparing the computation time of each of the scheduling strategy configurations to obtain a target scheduling strategy configuration that minimizes the computation time; Optimizing the operator to be optimized according to the target scheduling strategy configuration; The device also includes: A second mechanism switch module, used to enable the double buffer mechanism of the on-chip memory and the double buffer mechanism between the external storage and the on-chip memory; A second data segmentation module is used to perform data segmentation on the memory usage in a manner that maximizes the amount of data calculated at a single time by the hardware unit of the on-chip memory according to the basic information of the operator to be optimized and the on-chip memory, wherein the direction of the data segmentation includes the number of data blocks after segmentation, the number of channels, the height and the width; The device also includes: a cost model training module, which is used to: Obtain the initialized cost model and the sample basic information of the sample operator; Based on the sample basic information, generating a scheduling search space of the sample operator according to a preset scheduling template dimension; Searching for multiple scheduling strategy configurations of the sample operator in the scheduling search space, and sampling actual time consumption of each of the scheduling strategy configurations; Predicting the predicted time consumption of each of the scheduling strategy configurations using the initialized cost model; The cost model is trained based on the difference between the predicted time consumption and the actual time consumption.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images