Model reasoning optimization method, system, electronic device and storage medium

Abstract

The embodiments of the present application relate to the field of deep learning technology, and in particular to a method, system, electronic device and storage medium for inference optimization of a model, the method comprising: obtaining operator information of the model and hardware information of a central processing unit CPU; determining a code generation strategy according to the operator information of the model, the hardware information of the CPU and a preset strategy generation algorithm; wherein the code generation strategy comprises a scheduling code generation strategy and a calculation code generation strategy; according to the code generation strategy, a scheduling code and a calculation code are generated; the scheduling code and the calculation code are compiled and linked to obtain a compiled model file. The inference optimization method of the model provided in the embodiments of the present application can make full use of the computing power of the CPU, greatly improve the speed and efficiency of model inference, and realize targeted optimization of the model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning technology, and in particular to a method, system, electronic device and storage medium for optimizing the reasoning of a model. Background Art

[0002] With the rapid development of deep learning technology, more and more technologies can be realized by models, such as face recognition models, speech recognition models, etc. Service providers deploy trained models to provide inference services such as face recognition and speech recognition. Deep learning models can be deployed based on graphics processing units (GPUs) for inference services, which can meet the high throughput and low latency requirements of inference services. However, GPUs are very expensive, and the cost of deploying inference based on GPUs is very high and the power consumption is also very high. The deployment price of inference services based on central processing units (CPUs) is lower and the inference cost is also lower. When CPU inference can meet the requirements of throughput and latency, deployment based on CPUs is the first choice of service providers. In addition, on some devices that are not equipped with GPUs, inference services can only be deployed based on CPUs.

[0003] However, when technicians deploy deep learning models on the CPU for inference service deployment, they usually compile the model directly into the target file format and call the same general computing method as the training process to obtain the final inference result during calculation. Although this can meet the correctness requirements of inference, the inference speed is relatively slow, and it is impossible to perform targeted optimization on different models, nor can it give full play to the computing power of the CPU. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a model reasoning optimization method, system, electronic device and storage medium, which can fully utilize the computing power of the CPU, greatly improve the speed and efficiency of model reasoning, and achieve targeted optimization of the model.

[0005] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for reasoning optimization of a model, the method comprising: obtaining operator information of the model and hardware information of a central processing unit CPU; determining a code generation strategy based on the operator information of the model, the hardware information of the CPU and a preset strategy generation algorithm; wherein the code generation strategy includes a scheduling code generation strategy and a calculation code generation strategy; generating scheduling code and calculation code based on the code generation strategy; compiling and linking the scheduling code and the calculation code to obtain a compiled model file.

[0006] To achieve the above-mentioned purpose, an embodiment of the present application also provides a model reasoning optimization system, including a model analysis module, a hardware analysis module, a strategy generation module, a code generation module and a compiler module; the model analysis module is used to obtain the operator information of the model, and send the operator information of the model to the strategy generation module; the hardware analysis module is used to obtain the hardware information of the central processing unit CPU, and send the hardware information of the CPU to the strategy generation module; the strategy generation module is used to determine the code generation strategy according to the operator information of the model, the hardware information of the CPU and the preset strategy generation algorithm, and send the code generation strategy to the code generation module; wherein the code generation strategy includes a scheduling code generation strategy and a computing code generation strategy; the code generation module is used to generate a scheduling code and a computing code according to the code generation strategy, and send the scheduling code and the computing code to the compiler module; the compiler module is used to compile and link the scheduling code and the computing code to obtain a compiled model file.

[0007] To achieve the above-mentioned purpose, an embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned model inference optimization method.

[0008] To achieve the above objectives, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which implements the above-mentioned model reasoning optimization method when executed by a processor.

[0009] The model reasoning optimization method, system, electronic device and storage medium proposed in the present application first obtain the operator information of the model and the hardware information of the CPU, and determine the code generation strategy including the scheduling code generation strategy and the calculation code generation strategy according to the obtained operator information of the model, the hardware information of the CPU and the preset strategy generation algorithm, and then generate the scheduling code and the calculation code according to the code generation strategy, and compile and link the scheduling code and the calculation code to obtain the compiled model file. Considering that the operator information of different models is different, even the operators of the same model are different under different parameter situations, and the computing environment and computing power provided by different CPUs are also different, the use of a common computing method for all models and CPUs seriously affects the reasoning efficiency of the model and cannot give full play to the computing power of the CPU. This embodiment comprehensively considers the actual situation of the operator information of the model and the hardware information of the CPU, and formulates code generation strategies for different models and different CPUs in a targeted manner, thereby generating the scheduling code and calculation code that are most suitable for the model and the CPU, which can make full use of the computing power of the CPU, significantly reduce the reasoning delay, improve the reasoning throughput, and reduce the cost of reasoning, thereby greatly improving the speed and efficiency of model reasoning, and realizing targeted optimization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of a method for optimizing the reasoning of a model according to an embodiment of the present application. Figure 1 ;

[0011] Figure 2 It is a flowchart for determining a code generation strategy according to operator information of a model, hardware information of a CPU and a preset strategy generation algorithm in one embodiment of the present application;

[0012] Figure 3 It is a flowchart of determining the number of parallel threads and the parallel block strategy according to an embodiment of the present application according to the input dimension size, the output dimension size and the number of available threads of the CPU;

[0013] Figure 4 It is a flowchart of determining a parallel block strategy according to the number of parallel threads according to one embodiment of the present application;

[0014] Figure 5 This is a flow chart of a method for optimizing the reasoning of a model according to another embodiment of the present application. Figure 2 ;

[0015] Figure 6 This is a flow chart of a method for optimizing the reasoning of a model according to another embodiment of the present application. Figure 3 ;

[0016] Figure 7is a flowchart for obtaining operator information of a model according to one embodiment of the present application;

[0017] Figure 8 is a flowchart for determining a convolution parallel strategy to be used according to one embodiment of the present application;

[0018] Fig. 9 is a schematic diagram of a reasoning optimization system of a model provided in another embodiment of the present application;

[0019] Fig.10 It is a schematic diagram of the structure of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0021] With the development of deep learning technology, inference services including face recognition services and speech recognition services have great development prospects and a broad market. Inference service providers can deploy trained inference models to provide inference services to customers in a variety of ways, such as deploying on GPUs and CPUs. When inference service providers deploy deep learning models on CPUs for inference service deployment, they usually compile the models directly into target file formats and call the same general computing methods as the training process during calculations to obtain the final inference results.

[0022] Although the correctness requirement of reasoning can be met by calling the same general calculation method as the training process during calculation, the inventors of the present application have found that the above technical solution has at least the following problems:

[0023] (1) The neural network operators of deep learning models of different types and purposes are different. Calling general computing methods during the reasoning process cannot perform targeted optimization on different operators. The computing efficiency of the operators is low, and the reasoning efficiency of the model is also low.

[0024] (2) Even for the same neural network operator, it may be different in different parameter forms. If a general calculation method is used in the inference process, the calculation process of the operator in different parameter forms is the same, and it is impossible to perform targeted optimization on the operators in different parameter forms.

[0025] (3) Different CPUs have different computing capabilities. Simply using general computing methods cannot fully utilize the computing capabilities of the CPU.

[0026] In order to solve the above-mentioned technical problem of "the inference speed of the model is relatively slow, different models cannot be optimized in a targeted manner, and the computing power of the CPU cannot be fully utilized", an embodiment of the present application provides a model inference optimization method, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments are described using the server as an example. The implementation details of the model inference optimization method of this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0027] The specific process of the inference optimization method of the model in this embodiment can be as follows: Figure 1 As shown, including:

[0028] Step 101, obtain the operator information of the model and the hardware information of the CPU.

[0029] Exemplarily, when the server performs inference optimization of the model, it can first obtain the operator information of the model and the hardware information of the CPU, where the operator information of the model includes but is not limited to: input dimension size, output dimension size, and Batch-Size dimension size, etc.; the hardware information of the CPU includes but is not limited to: the instruction set set supported by the CPU, the number of available threads of the CPU, the number of single instruction multiple data registers of the CPU, the number of floating-point numbers that the single instruction multiple data register can process at one time, the number of bits of the operating system and the cache size, etc.

[0030] In an example, after the server obtains the operator information of the model from the model, it can recombine the operator information of the model, that is, perform standardization processing on the operator information of the model to generate standardized operator information of the model. Taking into account that different models may have different properties, formats, etc., this embodiment standardizes the operator information of the obtained model, which can improve the robustness and universality of the model's reasoning optimization method.

[0031] Step 102, determining a code generation strategy according to the operator information of the model, the hardware information of the CPU and a preset strategy generation algorithm.

[0032] Exemplarily, after obtaining the operator information of the model and the hardware information of the CPU, the server can determine the code generation strategy based on the operator information of the model, the hardware information of the CPU and a preset strategy generation algorithm, where the code generation strategy includes a scheduling code generation strategy and a calculation code generation strategy, and the preset strategy generation algorithm can be set by technical personnel in this field according to actual needs.

[0033] Exemplarily, the scheduling code generation strategy mainly describes the adjustment of the number of operator computing threads, the distribution of the number of threads in each dimension of the operator, the thread scheduling strategy of multi-threaded parallelism, the multi-threaded position determination strategy, the multi-threaded parallel blocking strategy and the operator scheduling and blocking strategy under each thread; the computing code generation strategy mainly describes the implementation method of the single instruction multiple input strategy, the order of loop calculations in each dimension, the computing core size and the computing core strategy, etc.

[0034] In one example, the server pre-stores the correspondence between the model's operator information, the CPU's hardware information, and the strategy generation algorithm. After obtaining the model's operator information and the CPU's hardware information, the server can call the pre-stored correspondence inside the server to determine the strategy generation algorithm to determine the code generation strategy.

[0035] Step 103: Generate scheduling code and calculation code according to the code generation strategy.

[0036] Exemplarily, after determining the scheduling code generation strategy and the computing code generation strategy, the server may generate the scheduling code according to the scheduling code generation strategy and generate the computing code according to the computing code generation strategy.

[0037] In one example, the scheduling code generated by the server is C++ code, and the generated calculation code is assembly code. Using this code format can further improve the speed and efficiency of model reasoning.

[0038] Step 104, compile and link the scheduling code and the calculation code to obtain a compiled model file.

[0039] Exemplarily, after the server generates the scheduling code and the calculation code, it can compile the scheduling code and the calculation code respectively, and link the compiled results to obtain a compiled model file.

[0040] In this embodiment, compared with the technical solution of directly compiling the model into a model file and using the same general computing method as the training process to realize the inference service during operator calculation, the server first obtains the operator information of the model and the hardware information of the CPU, and determines the code generation strategy including the scheduling code generation strategy and the calculation code generation strategy according to the obtained operator information of the model, the hardware information of the CPU and the preset strategy generation algorithm, and then generates the scheduling code and the calculation code according to the code generation strategy, and compiles and links the scheduling code and the calculation code to obtain the compiled model file. Considering that the operator information of different models is different, even if the operators of the same model are different, There are also differences under the same parameter situation, and different CPUs provide different computing environments and computing powers. Using a universal computing method for all models and CPUs seriously affects the reasoning efficiency of the model and cannot give full play to the computing power of the CPU. This embodiment comprehensively considers the actual situation of the operator information of the model and the hardware information of the CPU, and formulates code generation strategies for different models and different CPUs in a targeted manner, thereby generating the most suitable scheduling code and calculation code for the model and the CPU. The computing power of the CPU can be fully utilized, the reasoning latency can be significantly reduced, the reasoning throughput can be improved, and the reasoning cost can be reduced, thereby greatly improving the speed and efficiency of model reasoning and achieving targeted optimization of the model.

[0041] In one embodiment, the operator information of the model includes the input dimension size and the output dimension size, the hardware information of the CPU includes the instruction set set supported by the CPU, the number of available threads of the CPU, the number of single instruction multiple data registers of the CPU, and the number of floating point numbers that the single instruction multiple data registers can process at one time. According to the operator information of the model, the hardware information of the CPU and the preset strategy generation algorithm, the code generation strategy is determined, which can be achieved by Figure 2 The steps shown are implemented, specifically including:

[0042] Step 201, determining the number of parallel threads and the parallel block strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU.

[0043] Exemplarily, when determining the scheduling code generation strategy, the server can first determine the number of parallel threads and the parallel blocking strategy based on the dimension size and output dimension size in the operator information of the model and the number of available threads of the CPU in the hardware information of the CPU. The number of parallel threads can indicate whether the model requires multi-threaded parallelism, how many threads need to be used for multi-threaded parallelism, etc. The parallel blocking strategy can indicate which threads need to participate in multi-threaded parallelism, the size of each dimension that each thread participating in multi-threaded parallelism needs to process, etc.

[0044] Step 202, determining a scheduling code generation strategy according to the number of parallel threads, the parallel block strategy and the instruction set set supported by the CPU.

[0045] Exemplarily, after the server determines the number of parallel threads and the parallel blocking strategy, it can determine the scheduling code generation strategy based on the number of parallel threads and the parallel blocking strategy, combined with the instruction set set supported by the CPU in the CPU hardware information. If the server determines that multi-threaded parallelism is not required, it can call the preset single-threaded scheduling template to determine the scheduling code generation strategy. If the server determines that multi-threaded parallelism is required, it can call the preset multi-threaded scheduling template according to the number of parallel threads and the parallel blocking strategy to determine the scheduling code generation strategy. The preset single-threaded scheduling template and the preset multi-threaded scheduling template can be set by technicians in this field according to actual needs, and the embodiments of the present application do not specifically limit this.

[0046] Step 203, determining the computing core size and loop strategy according to the number of SIMD registers of the CPU and the number of floating point numbers that can be processed by the SIMD registers at one time.

[0047] Exemplarily, when determining the computing code generation strategy, the server can determine the computing core size and the loop strategy based on the number of single instruction multiple data registers of the CPU and the number of floating-point numbers that the single instruction multiple data registers can process at one time in the CPU hardware information, wherein the computing core size includes the computing core size of each dimension, the loop strategy includes the loop strategy of each dimension, and the order of loop calculation.

[0048] In an example, the operator information of the model includes the Batch-Size dimension. If the Batch-Size dimension is equal to 1, the calculation kernel size includes the calculation kernel size of the input dimension and the calculation kernel size of the output dimension. The server can determine the calculation kernel size by the following formula:

[0049]

[0050] Where M is the computing kernel size of the input dimension, N is the computing kernel size of the output dimension, L is the number of floating-point numbers that a SIMD register can process at one time, and R is the number of SIMD registers of the CPU.

[0051] In an example, the operator information of the model includes the Batch-Size dimension. If the Batch-Size dimension is equal to 1, the loop strategy includes the loop strategy of the input dimension and the loop strategy of the output dimension. The loop strategy determined by the server may be as shown in Table 1:

[0052] Table 1: Schematic diagram of the cycle strategy (I)

[0053] Number of single instruction multiple data registers in the CPU Input Dimension Cycle Strategy Output dimension loop strategy 16 (2L, L, L / 2, …, 1) (4，2，1) 8 (L, L / 2, ..., 1) (3，2，1) 4 (L, L / 2, ..., 1) 1

[0054] In Table 1, L is the number of floating-point numbers that a SIMD register can process at one time. Taking L equal to 8 and the number of SIMD registers of the CPU as 8 as an example, the input dimension loop strategy is (8, 4, 2, 1), and the input dimension size is 55. The server first loops according to 8. After 6 loops, the input dimension size is 7. The server then loops according to 4, loops once, and the input dimension size is 3. The server then loops according to 2, loops once, and the input dimension size is 1. The server then loops according to 1, loops once, and the input dimension size is 0. The server can end the loop.

[0055] Step 204 , determining a computing code generation strategy according to the computing core size, the loop strategy, and the instruction set set supported by the CPU.

[0056] In an example, the operator information of the model includes the Batch-Size dimension. If the Batch-Size dimension is equal to 1, after the server determines the computing core size and the loop strategy, it can determine the scheduling code generation strategy based on the computing core size and the loop strategy, combined with the instruction set set supported by the CPU in the CPU hardware information, that is, determine the outer loop framework, the inner loop framework, the calculation method of the matrix and vector product, and the result accumulation and storage method.

[0057] In this embodiment, when determining the scheduling code generation strategy, the server first determines the number of parallel threads and the parallel blocking strategy. When determining the calculation code generation strategy, the server first determines the calculation core size and the loop strategy, that is, blocking, multi-threaded parallelism and looping are performed during the reasoning process, thereby further improving the speed and efficiency of model reasoning and enhancing the user experience.

[0058] In one embodiment, the server determines the number of parallel threads and the parallel block strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU, which can be achieved by Figure 3 The steps shown are implemented, specifically including:

[0059] Step 301, determine whether the output dimension size is greater than a first preset threshold and less than a second preset threshold, if yes, execute step 302, otherwise, execute step 303.

[0060] Step 302: Determine the number of parallel threads as the number of available threads of the CPU, and determine a parallel block strategy according to the number of parallel threads.

[0061] Exemplarily, when the server determines the number of parallel threads and the parallel blocking strategy based on the input dimension size, the output dimension size and the number of available threads of the CPU, it first determines whether the output dimension size is greater than the first preset threshold and less than the second preset threshold. If the output dimension size is greater than the first preset threshold and less than the second preset threshold, the server determines the number of parallel threads to be the number of available threads of the CPU, and determines the parallel blocking strategy based on the number of parallel threads, that is, all threads available to the CPU participate in multi-threaded parallelism, wherein the first preset threshold and the second preset threshold can be set by technicians in this field according to actual needs, and the embodiments of the present application do not specifically limit this.

[0062] Step 303 , determine whether the product of the input dimension size and the output dimension size is less than a third preset threshold, if yes, execute step 304 , otherwise, execute step 305 .

[0063] Step 304, determine that the number of parallel threads is 1.

[0064] Exemplarily, if the output dimension size is not greater than the first preset threshold, or not less than the second preset threshold, the server continues to determine whether the product of the input dimension size and the output dimension size is less than the third preset threshold. If the server determines that the product of the input dimension size and the output dimension size is less than the third preset threshold, there is no need to perform multi-threaded parallelism. The server determines that the number of parallel threads is 1, that is, single-threaded operation, where the third preset threshold can be set by technical personnel in this field according to actual needs.

[0065] Step 305, calculate the ratio of the output dimension size to the number of available threads of the CPU.

[0066] Step 306, determining the number of parallel threads according to the ratio of the output dimension size to the number of available threads of the CPU, and determining the parallel blocking strategy according to the number of parallel threads.

[0067] Exemplarily, if the server determines that the product of the input dimension size and the output dimension size is greater than or equal to a third preset threshold, the server can calculate the ratio of the output dimension size to the number of available threads of the CPU, determine the number of parallel threads based on the ratio of the output dimension size to the number of available threads of the CPU and a preset formula, and determine the parallel blocking strategy based on the number of parallel threads.

[0068] In this embodiment, the server determines the number of parallel threads and the parallel blocking strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU, which can give full play to the computing power of the CPU and better meet the actual needs of the model, thereby further improving the reasoning efficiency of the model.

[0069] In one embodiment, the server determines the parallel partitioning strategy according to the number of parallel threads, which can be done as follows: Figure 4The steps shown are implemented, specifically including:

[0070] Step 401, determine whether the output dimension size p is divisible by the number of parallel threads q, if yes, execute step 402, otherwise, execute step 403.

[0071] Step 402, determine that the output dimension size that each thread needs to process is p / q.

[0072] Step 403, determine that the output dimension size that each thread in the q-1 threads needs to process is 1+p / q, and the output dimension size that the remaining thread needs to process is p-(q-1)×(1+p / q).

[0073] Exemplarily, q is an integer greater than 1, and p is an integer greater than or equal to q. After the server determines the number of parallel threads q, it determines whether the output dimension size p is divisible by the number of parallel threads q. If p is divisible by p, the server determines that the output dimension size that each thread among the q threads participating in multi-threaded parallelism needs to process is p / q; if p is not divisible by q, the server determines that the output dimension size that each thread among the q-1 threads among the q threads participating in multi-threaded parallelism needs to process is 1+p / q, and the output dimension size that the remaining thread needs to process is p-(q-1)×(1+p / q).

[0074] Another embodiment of the present application provides a method for optimizing the reasoning of a model. In this embodiment, the operator information of the model includes the Batch-Size dimension. In this embodiment, the Batch-Size dimension is equal to 1. The implementation details of the method for optimizing the reasoning of the model of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution. The specific process of the method for optimizing the reasoning of the model of this embodiment can be as follows Figure 5 As shown, including:

[0075] Step 501, obtain the operator information of the model and the hardware information of the CPU.

[0076] Step 502, determining the number of parallel threads and the parallel block strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU.

[0077] Step 503: Determine a scheduling code generation strategy according to the number of parallel threads, the parallel block strategy, and the instruction set set supported by the CPU.

[0078] Step 504 , determining the computing core size and loop strategy according to the number of SIMD registers of the CPU and the number of floating point numbers that can be processed by the SIMD registers at one time.

[0079] Step 505 , determining a computing code generation strategy according to the computing core size, the loop strategy, and the instruction set set supported by the CPU.

[0080] Step 506: Generate a scheduling code and a computing code according to the scheduling code generation strategy and the computing code generation strategy.

[0081] Step 507, compile and link the scheduling code and the calculation code to obtain a compiled model file.

[0082] Another embodiment of the present application provides a method for optimizing the reasoning of a model. In this embodiment, the operator information of the model includes the size of the Batch-Size dimension. In this embodiment, the size of the Batch-Size dimension is greater than 1. The implementation details of the method for optimizing the reasoning of the model of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution. The specific process of the method for optimizing the reasoning of the model of this embodiment can be as follows Figure 6 As shown, including:

[0083] Step 601, obtaining operator information of the model and hardware information of the CPU.

[0084] Among them, step 601 is substantially the same as step 101 and will not be described again here.

[0085] Step 602, determine the number of parallel threads and the parallel block strategy according to the input dimension size, the output dimension size, the Batch-Size dimension size and the number of available threads of the CPU.

[0086] In one example, the parallel blocking strategy includes a parallel blocking strategy of an output dimension and a parallel blocking strategy of a Batch-Size dimension.

[0087] Step 603, determining a scheduling code generation strategy according to the number of parallel threads, the parallel block strategy and the instruction set set supported by the CPU.

[0088] Among them, step 603 is substantially the same as step 202 and will not be described again here.

[0089] Step 604 , determining the computing core size and loop strategy according to the number of SIMD registers of the CPU and the number of floating point numbers that can be processed by the SIMD registers at one time.

[0090] In an example, when the Batch-Size dimension size is greater than 1, the calculation kernel size includes the calculation kernel size of the output dimension, the calculation kernel size of the output dimension, and the calculation kernel size of the Batch-Size dimension. The server can determine the calculation kernel size by the following formula:

[0091]

[0092] Where M is the computing kernel size of the input dimension, K is the computing kernel size of the Batch-Size dimension, N is the computing kernel size of the output dimension, L is the number of floating-point numbers that a single instruction multiple data register can process at one time, and R is the number of single instruction multiple data registers of the CPU.

[0093] In an example, if the Batch-Size dimension is greater than 1, the loop strategy includes a loop strategy of the input dimension, a loop strategy of the output dimension, and a loop strategy of the Batch-Size dimension. The loop strategy determined by the server may be as shown in Table 2:

[0094] Table 2: Cycle strategy diagram (II)

[0095]

[0096] In Table 2, L is the number of floating-point numbers that can be processed by a SIMD register at one time. When the remaining size of the output dimension is less than 2L or L, the server can perform processing such as padding with zeros.

[0097] Step 605, obtain the memory arrangement of the weight matrix of the model, and determine whether the input dimension is the dominant dimension.

[0098] Step 606: When the input dimension is the dominant dimension, determine the weight matrix rearrangement strategy.

[0099] Exemplarily, when determining the calculation code generation strategy, the server first obtains the memory layout of the model's weight matrix, and determines whether the memory layout of the model's weight matrix is ​​dominated by the input dimension. If the memory layout of the model's weight matrix is ​​dominated by the input dimension, the server needs to rearrange the model's weight matrix and determine the weight matrix rearrangement strategy.

[0100] Step 607 , determining a computing code generation strategy according to the computing core size, the loop strategy, the instruction set set supported by the CPU, and the weight matrix rearrangement strategy.

[0101] Exemplarily, after the server determines the computing core size, loop strategy and weight matrix reordering strategy, it can determine the scheduling code generation strategy based on the computing core size, loop strategy and weight matrix reordering strategy, combined with the instruction set set supported by the CPU in the CPU hardware information, that is, determine the outer loop framework, middle loop framework, inner loop framework, matrix and vector product calculation method, and result accumulation and storage method.

[0102] Step 608: Generate scheduling code and calculation code according to the code generation strategy.

[0103] Step 609, compile and link the scheduling code and the calculation code to obtain a compiled model file.

[0104] Among them, step 608 to step 609 are substantially the same as step 103 to step 104 and are not described again here.

[0105] In this embodiment, the operator information of the model also includes the Batch-Size dimension size. If the Batch-Size dimension size is greater than 1, before determining the computing code generation strategy based on the computing core size, the loop strategy and the instruction set set supported by the CPU, it includes: obtaining the memory arrangement form of the weight matrix of the model, and judging whether the input dimension is the dominant dimension; when the input dimension is the dominant dimension, determining the weight matrix rearrangement strategy; determining the computing code generation strategy based on the computing core size, the loop strategy and the instruction set set supported by the CPU, including: determining the computing code generation strategy based on the computing core size, the loop strategy, the instruction set set supported by the CPU and the weight matrix rearrangement strategy. When the memory arrangement form of the weight matrix is ​​that the input dimension is the dominant dimension, the data reading of the model is relatively slow. The embodiment of the present application rearranges the memory arrangement form of the weight matrix to increase the data reading speed, thereby further improving the reasoning speed and efficiency of the model.

[0106] In one embodiment, the operator of the model is a two-dimensional convolution operator, and the server obtains the operator information of the model by: Figure 7 The steps shown are implemented, specifically including:

[0107] Step 701, obtaining information of a two-dimensional convolution operator.

[0108] Exemplarily, the information of the two-dimensional convolution operator includes but is not limited to the convolution kernel size in the height direction, the convolution kernel size in the width direction, the step size in the height direction, the step size in the width direction and the Padding value of the two-dimensional convolution operator.

[0109] Step 702, when at least one of the convolution kernel size in the height direction, the convolution kernel size in the width direction, the step size in the height direction, the step size in the width direction, and the Padding value of the two-dimensional convolution operator is not equal to 1, determine the image rearrangement strategy for the two-dimensional convolution operator.

[0110] Step 703: According to the image rearrangement strategy, the two-dimensional convolution operator is converted into a fully connected operator, and the information of the fully connected operator is obtained.

[0111] Exemplarily, after the server obtains the information of the two-dimensional convolution operator, it can determine whether the convolution kernel size in the height direction, the convolution kernel size in the width direction, the step size in the height direction, the step size in the width direction, and the Padding value of the two-dimensional convolution operator are all equal to 1. If at least one of the convolution kernel size in the height direction, the convolution kernel size in the width direction, the step size in the height direction, the step size in the width direction, and the Padding value of the two-dimensional convolution operator is not equal to 1, the server needs to rearrange the image of the operator and use a preset image rearrangement strategy to arrange the two-dimensional convolution operator into an ordinary fully connected operator so that the server can subsequently use the information of the rearranged fully connected operator for calculation.

[0112] In this embodiment, when it is determined that the operator of the model is a two-dimensional convolution operator, an image rearrangement strategy for the two-dimensional convolution operator can be determined to rearrange the two-dimensional convolution operator into an ordinary fully connected operator, thereby greatly improving the robustness and universality of the model's reasoning optimization method and better meeting the actual needs of users.

[0113] In one embodiment, the information of the two-dimensional convolution operator includes the Batch-Size dimension size, the height of the output dimension, the width of the output dimension, the number of channels of the output dimension, and the number of channels of the input dimension. After obtaining the information of the two-dimensional convolution operator, the server can perform the following operations: Figure 8 The steps shown determine the convolutional parallelism strategy used, including:

[0114] Step 801, determine whether the Batch-Size dimension size is equal to 1, if so, execute step 802, otherwise, execute step 803.

[0115] Step 802, using the first convolution parallel strategy.

[0116] Exemplarily, the multi-threaded parallelism of the two-dimensional convolution operator is different from the multi-threaded parallelism of the ordinary fully connected operator. The server first determines whether the Batch-Size dimension is equal to 1. If the Batch-Size dimension is equal to 1, the two-dimensional convolution operator requires a lot of time in image rearrangement, input matrix calculation and weight matrix calculation. The server determines to use the first convolution parallel strategy for the operator, wherein the first convolution parallel strategy is used for multi-threaded parallelism in image rearrangement, input matrix calculation and weight matrix calculation.

[0117] Step 803, determine whether the information of the two-dimensional convolution operator meets the preset second convolution parallel strategy usage conditions, if yes, execute step 804, otherwise, execute step 802.

[0118] Step 804, using the second convolution parallel strategy.

[0119] Exemplarily, if the server determines that the Batch-Size dimension of the two-dimensional convolution operator is greater than 1, it continues to determine whether the information of the two-dimensional convolution operator meets the preset second convolution parallel strategy usage condition, wherein the preset second convolution parallel strategy usage condition can be expressed by the following formula:

[0120] oh×ow×oc <mb∪oh×ow×oc×ic<threshold

[0121] In the formula, oh is the height of the output dimension, ow is the width of the output dimension, oc is the number of channels in the output dimension, mb is the Batch-Size dimension size, ic is the number of channels in the input dimension, threshold is the fourth preset threshold, and the fourth preset threshold can be set by technical personnel in this field according to actual needs. If the information of the two-dimensional convolution operator meets the preset second convolution parallel strategy usage conditions, it means that the operator takes a lot of time in the loop and convolution calculation. The server uses the second convolution parallel strategy for the operator, and the second convolution parallel strategy is used to perform multi-threaded parallelism in the loop and convolution calculation.

[0122] In this embodiment, when performing two-dimensional convolution operator calculations, the server will evaluate the approximate time consumption of the operator performing multi-threaded parallelism during image rearrangement, input matrix calculations, and weight matrix calculations (i.e., the first convolution parallel strategy) and the operator performing multi-threaded parallelism during loop and convolution calculations (i.e., the second convolution parallel strategy), and select the convolution parallel strategy with less calculation time based on the above judgment process to accelerate the two-dimensional convolution operator calculations and improve the speed.

[0123] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0124] Another embodiment of the present application relates to a model reasoning optimization system. The details of the model reasoning optimization system of this embodiment are described in detail below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this example. Fig. 9 It is a schematic diagram of the inference optimization system of the model described in this embodiment, including: a model analysis module 901, a hardware analysis module 902, a strategy generation module 903, a code generation module 904 and a compiler module 905.

[0125] The model analysis module 901 is used to obtain operator information of the model and send the operator information of the model to the strategy generation module 903 .

[0126] The hardware analysis module 902 is used to obtain the hardware information of the CPU and send the hardware information of the CPU to the strategy generation module 903 .

[0127] The strategy generation module 903 is used to determine the code generation strategy according to the operator information of the model, the hardware information of the CPU and the preset strategy generation algorithm, and send the code generation strategy to the code generation module 904; wherein the code generation strategy includes a scheduling code generation strategy and a calculation code generation strategy.

[0128] The code generation module 904 is used to generate a scheduling code and a calculation code according to a code generation strategy, and send the scheduling code and the calculation code to the compiler module 905 .

[0129] The compiler module 905 is used to compile and link the scheduling code and the calculation code to obtain a compiled model file.

[0130] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0131] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0132] Another embodiment of the present application relates to an electronic device, such as Fig.10 As shown, it includes: at least one processor 1001; and a memory 1002 that is communicatively connected to the at least one processor 1001; wherein the memory 1002 stores instructions that can be executed by the at least one processor 1001, and the instructions are executed by the at least one processor 1001 so that the at least one processor 1001 can execute the inference optimization method of the model in the above-mentioned embodiments.

[0133] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0134] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0135] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0136] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0137] Those skilled in the art will appreciate that the above-mentioned embodiments are specific examples for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for optimizing the reasoning of a model, characterized in that: include: Obtaining operator information of the model and hardware information of a central processing unit (CPU), wherein the operator information of the model includes input dimension size and output dimension size, and the hardware information of the CPU includes an instruction set set supported by the CPU, the number of available threads of the CPU, the number of single instruction multiple data registers of the CPU, and the number of floating-point numbers that the single instruction multiple data register can process at one time; Determine the number of parallel threads and the parallel block strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU; Determine a scheduling code generation strategy according to the number of parallel threads, the parallel block strategy and the instruction set set supported by the CPU; Determine a computing core size and a loop strategy according to the number of single instruction multiple data registers of the CPU and the number of floating point numbers that can be processed by the single instruction multiple data registers at one time; Determining a computing code generation strategy according to the computing core size, the loop strategy, and an instruction set set supported by the CPU; Generate a scheduling code according to the scheduling code generation strategy, and generate a calculation code according to the calculation code generation strategy; The scheduling code and the calculation code are compiled and linked to obtain a compiled model file.

2. The inference optimization method of the model according to claim 1, characterized in that: The determining the number of parallel threads and the parallel blocking strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU comprises: Determine whether the output dimension size is greater than a first preset threshold and less than a second preset threshold; When the output dimension size is greater than a first preset threshold and less than a second preset threshold, determining the number of parallel threads as the number of available threads of the CPU, and determining a parallel block strategy according to the number of parallel threads; When the output dimension size is not greater than the first preset threshold, or not less than the second preset threshold, determining whether the product of the input dimension size and the output dimension size is less than a third preset threshold; When the product is less than a third preset threshold, determining the number of parallel threads to be 1; When the product is greater than or equal to a third preset threshold, calculating a ratio of the output dimension size to the number of available threads of the CPU; The number of parallel threads is determined according to the ratio, and a parallel blocking strategy is determined according to the number of parallel threads.

3. The inference optimization method of the model according to claim 2, characterized in that: Determining the parallel block strategy according to the number of parallel threads includes: Determine whether the output dimension size p is divisible by the number of parallel threads q; wherein q is an integer greater than 1, and p is an integer greater than or equal to q; When the output dimension size p is divisible by the number of parallel threads q, determine that the output dimension size that each thread needs to process is p / q; When the output dimension size p cannot be divided by the number of parallel threads q, it is determined that the output dimension size that each of the q-1 threads needs to process is 1+p / q, and the output dimension size that the remaining thread needs to process is p-(q-1)×(1+p / q).

4. The inference optimization method of the model according to claim 1 or 2, characterized in that: The calculation kernel size includes the calculation kernel size of the input dimension and the calculation kernel size of the output dimension, and the loop strategy includes the loop strategy of the input dimension and the loop strategy of the output dimension; The calculation kernel size is determined by the following formula: In the formula, M is the computing core size of the input dimension, N is the computing core size of the output dimension, L is the number of floating-point numbers that the SIMD register can process at one time, and R is the number of SIMD registers of the CPU.

5. The inference optimization method of the model according to claim 1, characterized in that: The operator information of the model also includes the Batch-Size dimension size. If the Batch-Size dimension size is greater than 1, before determining the computing code generation strategy according to the computing core size, the loop strategy, and the instruction set set supported by the CPU, it includes: Obtaining the memory arrangement of the weight matrix of the model, and determining whether the input dimension is the dominant dimension; In the case where the input dimension is the dominant dimension, determining a weight matrix rearrangement strategy; The determining of the computing code generation strategy according to the computing core size, the loop strategy and the instruction set set supported by the CPU includes: A computing code generation strategy is determined according to the computing core size, the loop strategy, the instruction set set supported by the CPU, and the weight matrix rearrangement strategy.

6. The inference optimization method of the model according to claim 1, characterized in that: The operator information of the model also includes the Batch-Size dimension size. If the Batch-Size dimension size is greater than 1, the parallel blocking strategy includes the parallel blocking strategy of the output dimension and the parallel blocking strategy of the Batch-Size dimension.

7. The inference optimization method of the model according to claim 1, characterized in that: If the operator of the model is a two-dimensional convolution operator, the acquiring of the operator information of the model includes: Acquire information of the two-dimensional convolution operator; wherein the information of the two-dimensional convolution operator includes a convolution kernel size in a height direction, a convolution kernel size in a width direction, a step size in a height direction, a step size in a width direction, and a padding value of the two-dimensional convolution operator; When at least one of the convolution kernel size in the height direction, the convolution kernel size in the width direction, the step size in the height direction, the step size in the width direction, and the Padding value of the two-dimensional convolution operator is not equal to 1, determining an image rearrangement strategy for the two-dimensional convolution operator; According to the image rearrangement strategy, the two-dimensional convolution operator is converted into a fully connected operator, and information of the fully connected operator is obtained.

8. The inference optimization method of the model according to claim 7, characterized in that: The information of the two-dimensional convolution operator includes the Batch-Size dimension size, the height of the output dimension, the width of the output dimension, the number of channels in the output dimension, and the number of channels in the input dimension; After acquiring the information of the two-dimensional convolution operator, the method further includes: Determine whether the Batch-Size dimension is greater than 1; When the Batch-Size dimension is not greater than 1, a first convolution parallel strategy is used; wherein the first convolution parallel strategy is used to perform multi-threaded parallelism during image rearrangement, input matrix calculation, and weight matrix calculation; When the Batch-Size dimension is greater than 1, determine whether the information of the two-dimensional convolution operator meets the preset second convolution parallel strategy use condition; wherein the preset second convolution parallel strategy use condition includes the following formula: oh×ow×oc <mb∪oh×ow×oc×ic<threshold Wherein, oh is the height of the output dimension, ow is the width of the output dimension, oc is the number of channels of the output dimension, mb is the size of the Batch-Size dimension, ic is the number of channels of the input dimension, and threshold is the fourth preset threshold; When the information of the two-dimensional convolution operator meets the preset conditions for using the second convolution parallel strategy, the second convolution parallel strategy is used; wherein the second convolution parallel strategy is used to perform multi-threaded parallelism during loop and convolution calculations.

9. The inference optimization method of the model according to any one of claims 1 to 8, characterized in that: The scheduling code is a C++ code, and the calculation code is an assembly code.

10. A model reasoning optimization system, characterized in that: It includes model analysis module, hardware analysis module, strategy generation module, code generation module and compiler module; The model analysis module is used to obtain operator information of the model and send the operator information of the model to the strategy generation module, wherein the operator information of the model includes input dimension size and output dimension size; The hardware analysis module is used to obtain hardware information of a central processing unit (CPU), and send the hardware information of the CPU to the strategy generation module, wherein the hardware information of the CPU includes an instruction set set supported by the CPU, the number of available threads of the CPU, the number of single instruction multiple data registers of the CPU, and the number of floating point numbers that the single instruction multiple data registers can process at one time; The strategy generation module is used to determine the number of parallel threads and the parallel blocking strategy according to the input dimension size, the output dimension size and the number of available threads of the CPU; determine the scheduling code generation strategy according to the number of parallel threads, the parallel blocking strategy and the instruction set set supported by the CPU; determine the computing core size and the loop strategy according to the number of single instruction multiple data registers of the CPU and the number of floating-point numbers that the single instruction multiple data registers can process at one time; determine the computing code generation strategy according to the computing core size, the loop strategy and the instruction set set supported by the CPU; send the scheduling code generation strategy and the computing code generation strategy to the code generation module; The code generation module is used to generate a scheduling code according to the scheduling code generation strategy, generate a computing code according to the computing code generation strategy, and send the scheduling code and the computing code to the compiler module; The compiler module is used to compile and link the scheduling code and the calculation code to obtain a compiled model file.

11. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the inference optimization method of the model as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for inference optimization of the model according to any one of claims 1 to 9 is implemented.

Images