Compilation method and apparatus for neural network model, inference method and apparatus for neural network model, and device and medium

By converting dot product operations, matrix multiplication, etc. in the neural network model into convolutional operations, the inference process of the neural network processor is accelerated, and the problem of low inference speed based on the Transformer model is solved, and scenarios such as autonomous driving and medical image processing are improved efficiency.

WO2024239971A1PCT designated stage expired Publication Date: 2024-11-28BEIJING HORIZON INFORMATION TECH CO LTD

Patent Information

Application Number
PCT/CN2024/092000
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-05-09
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The neural network model based on network structures such as Transformer has a low inference speed when performing tasks such as image processing and text processing, which is mainly due to inefficiency that includes dot product operations, matrix multiplication operations and linear layers.

Method used

By converting these preset operations into convolution operations supported by the neural network processor during the compilation phase, a target instruction sequence is generated to accelerate model inference and accelerated using the convolutional neural network processor.

Benefits of technology

It improves the inference speed of neural network models and improves the real-time and efficiency of scenarios such as autonomous driving and medical image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024092000_28112024_PF_FP_ABST
    Figure CN2024092000_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the embodiments of the present disclosure are a compilation method and apparatus for a neural network model, an inference method and apparatus for a neural network model, and a device and a medium. The compilation method comprises: acquiring a first model corresponding to a neural network model to be subjected to compilation; using, as a target network layer, any one of at least one network layer which includes a preset operation in the first model, and converting the preset operation of the target network layer into a convolution operation which is supported by a neural network processor, so as to obtain a converted first network layer, wherein the preset operation is an operation comprising a multiply–accumulate operation; on the basis of the converted first network layer, obtaining an optimization model corresponding to the first model; and according to the optimization model, generating a target instruction sequence that is executable by the neural network processor and corresponds to said neural network model. In the embodiments of the present disclosure, a preset operation comprising a multiply–accumulate operation may be implemented by means of a convolution operation, such that the preset operation of a model is accelerated by means of a neural network processor, thereby facilitating an improvement in the inference speed of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Compilation method, inference method, device, equipment and medium for neural network model

[0001] This disclosure claims priority to Chinese patent application number CN202310597929.6, filed with the State Intellectual Property Office on May 24, 2023, entitled “Compilation method, inference method, device, equipment and medium for neural network model”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence technology, and in particular to a compilation method, reasoning method, apparatus, device, and medium for a neural network model. Background Art

[0003] In recent years, neural network models based on network structures such as Transformer (hereinafter referred to as models) have been widely used in various fields or scenarios that require image processing, text processing, speech processing, etc., such as visual perception scenarios in autonomous driving, medical image processing scenarios in the medical field, speech recognition scenarios in human-computer interaction, office text recognition scenarios, etc. In order to improve the inference speed of neural network models, it is usually necessary to accelerate the inference process of neural network models through neural network processors. Neural network processors are mainly used to accelerate the processing of convolutional neural networks (CNNs). However, models based on network structures such as Transformer usually also include operations such as dot product operations, matrix multiplication operations, and linear operations in linear layers. These operations can easily lead to lower model inference speeds.

[0004] Summary of the Invention

[0005] Embodiments of the present disclosure provide a compilation method, inference method, apparatus, device, and medium for a neural network model, which can improve the model inference speed.

[0006] One aspect of the present disclosure provides a method for compiling a neural network model, including: obtaining a first model corresponding to a neural network model to be compiled; taking any network layer of at least one network layer in the first model that includes a preset operation as a target network layer, converting the preset operation of the target network layer into a convolution operation supported by a neural network processor, and obtaining a converted first network layer, wherein the preset operation is an operation including a multiplication-accumulation operation; based on the converted first network layer, obtaining an optimized model corresponding to the first model; and generating a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled according to the optimized model.

[0007] Another aspect of the present disclosure provides an inference method for a neural network model, including: obtaining an input tensor corresponding to the neural network model and a target instruction sequence corresponding to the neural network model; the target instruction sequence is obtained based on the compilation method of the neural network model described in any of the above embodiments; transmitting the input tensor and the target instruction sequence to a neural network processor; and processing the input tensor based on the execution of the target instruction sequence by the neural network processor to obtain an inference result corresponding to the input tensor.

[0008] In another aspect of the present disclosure, a compilation device for a neural network model is provided, comprising: a first acquisition module for acquiring a first model corresponding to the neural network model to be compiled; a first processing module for taking any network layer of at least one network layer containing a preset operation in the first model as a target network layer, converting the preset operation of the target network layer into a convolution operation supported by a neural network processor, and obtaining a converted first network layer, wherein the preset operation is an operation including a multiplication-accumulation operation; a second processing module for obtaining an optimized model corresponding to the first model based on the converted first network layer; and a third processing module for generating a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled according to the optimized model.

[0009] Another aspect of the present disclosure provides an inference device for a neural network model, comprising: a second acquisition module, configured to acquire an input tensor corresponding to the neural network model and a target instruction sequence corresponding to the neural network model; the target instruction sequence is obtained based on the compilation method of the neural network model described in any of the above embodiments; a transmission module, configured to transmit the input tensor and the target instruction sequence to a neural network processor; the neural network processor, configured to execute the target instruction sequence to process the input tensor and obtain an inference result corresponding to the input tensor.

[0010] In another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the compilation method of the neural network model described in any of the above embodiments of the present disclosure, or execute the reasoning method of the neural network model described in any of the above embodiments of the present disclosure.

[0011] Another aspect of the present disclosure provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method for compiling a neural network model described in any of the above embodiments of the present disclosure.

[0012] In another aspect of the present disclosure, a computer program product is provided. When the instructions in the computer program product are executed by a processor, the compilation method of the neural network model provided by any embodiment of the present disclosure is executed, or the reasoning method of the neural network model provided by any embodiment of the present disclosure is executed.

[0013] Based on the compilation method, inference method, apparatus, device and medium of the neural network model provided by the above-mentioned embodiments of the present disclosure, it is possible to convert preset operations including dot product operations, matrix multiplication operations, linear operations, and the like, including multiplication-accumulation operations, into convolution operations supported by the neural network processor during the model compilation process, so that the generated target instruction sequence can implement the preset operations through convolution operations, thereby enabling the above-mentioned preset operations of the model to be accelerated by the neural network processor used to accelerate the convolutional neural network, which helps to improve the model inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a flow chart of a method for compiling a neural network model according to an exemplary embodiment of the present disclosure;

[0015] FIG2 is a flow chart of a method for compiling a neural network model according to an exemplary embodiment of the present disclosure;

[0016] FIG3 is a flow chart of a method for compiling a neural network model according to another exemplary embodiment of the present disclosure;

[0017] FIG4 is a flowchart of a method for compiling a neural network model provided by yet another exemplary embodiment of the present disclosure;

[0018] FIG5 is a schematic diagram of a data structure transformation principle of a first input tensor provided by an exemplary embodiment of the present disclosure;

[0019] FIG6 is a schematic diagram of a data structure transformation principle of a second input tensor provided by an exemplary embodiment of the present disclosure;

[0020] FIG7 is a schematic diagram of a data structure transformation principle of a second input tensor provided by another exemplary embodiment of the present disclosure;

[0021] FIG8 is a schematic diagram of a convolution operation principle of a third input tensor and a convolution weight tensor provided by an exemplary embodiment of the present disclosure;

[0022] FIG9 is a flowchart of an inference method of a neural network model provided by an exemplary embodiment of the present disclosure;

[0023] FIG10 is a schematic diagram of the structure of a compiling device for a neural network model provided by an exemplary embodiment of the present disclosure;

[0024] FIG11 is a schematic structural diagram of a compiling device for a neural network model provided by another exemplary embodiment of the present disclosure;

[0025] FIG12 is a schematic structural diagram of a first processing module 502 provided by an exemplary embodiment of the present disclosure;

[0026] FIG13 is a schematic diagram of the structure of an inference device of a neural network model provided by an exemplary embodiment of the present disclosure;

[0027] FIG14 is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0029] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0030] Overview of the Disclosure

[0031] In the process of realizing the present disclosure, the inventors discovered that in recent years, neural network models based on network structures such as Transformer (hereinafter referred to as models) have been widely used in various fields or scenarios that require image processing, text processing, speech processing, etc., such as visual perception scenarios in autonomous driving, medical image processing scenarios in the medical field, speech recognition scenarios in human-computer interaction, office text recognition scenarios, and so on. In order to improve the reasoning speed of the neural network model, it is usually necessary to accelerate the reasoning process of the neural network model through a neural network processor. The neural network processor is mainly used to accelerate the processing of convolutional neural networks (CNN). However, in models based on network structures such as Transformer, dot product operations, matrix multiplication operations, linear operations of linear layers, and other operations are usually also included. These operations easily lead to a low model reasoning speed.

[0032] Exemplary Overview

[0033] FIG1 is an exemplary application scenario of the compilation method of the neural network model provided by the present disclosure.

[0034] For neural network models with preset operations such as dot product operations, matrix multiplication operations, and linear operations of linear layers, such as target detection models, semantic segmentation models, image classification models, and other image processing models used for image processing of the exterior environment or smart cockpit in autonomous driving scenarios, and speech recognition models used for in-vehicle user voice interaction, etc., the compilation method of the neural network model disclosed herein can be used as the neural network model to be compiled. By obtaining a first model corresponding to the neural network model to be compiled during the compilation phase, the first model can be optimized, and at least one preset operation contained in the first model can be converted into a convolution operation supported by the neural network processor to obtain an optimized model. Then, a target instruction sequence executable by the neural network processor can be generated based on the optimized model, so that the target instruction sequence can implement the preset operation through the convolution operation, thereby accelerating the above-mentioned preset operation of the model through the neural network processor used to accelerate the convolutional neural network, which helps to improve the model inference speed. As shown in Figure 1, the compilation of the neural network model can be achieved through a compiler. The compiler can include a front-end, a mid-end and a back-end. The front-end, the mid-end and the back-end can respectively include different modular compiler components and tool chains. The front-end can convert the neural network model to be compiled from the programming language into an intermediate representation model, the mid-end can optimize the intermediate representation model to obtain an optimized model, and the back-end can convert the optimized model into a machine-executable target instruction sequence. The target instruction sequence can include an instruction sequence executable by a neural network processor, and the neural network processor can complete real-time reasoning of the neural network by executing these instructions. The compilation method of the neural network model disclosed in the present invention can be implemented at any stage of the front-end, the mid-end and the back-end of the compilation process, and is not specifically limited. That is, the first model can be a model described by the programming language of the front-end, or a model represented in the intermediate representation, or a model described by the initial instruction sequence generated by the back-end. It can be set according to actual needs.

[0035] The compilation method of the neural network model disclosed in the present invention can be applied to accelerate scenarios or fields involving the above-mentioned preset operations on any neural network processor, such as autonomous driving scenarios, medical image processing scenarios, voice interaction scenarios, etc., without limitation to specific scenarios and fields.

[0036] Exemplary Methods

[0037] FIG2 is a flow chart of a method for compiling a neural network model according to an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as servers, terminal devices, and the like. As shown in FIG2 , the method according to the embodiment of the present disclosure may include the following steps:

[0038] Step 201: Obtain a first model corresponding to the neural network model to be compiled.

[0039] The neural network model to be compiled can be a trained neural network model that includes at least one network layer containing preset operations, such as a trained Transformer-based object detection model, semantic segmentation model, etc. The specific type and function of the neural network model are not limited. The first model can be a model described in the programming language corresponding to the neural network model to be compiled, or a model represented by the intermediate representation corresponding to the neural network model to be compiled, or a model described by the initial instruction sequence corresponding to the generated neural network model to be compiled. Taking the first model as an intermediate representation model as an example, the neural network model to be compiled can be converted from the programming language to the intermediate representation by a neural network compiler to obtain the first model of the intermediate representation corresponding to the neural network model to be compiled. The specific method of converting from the programming language to the intermediate representation can adopt any feasible method according to actual needs and will not be described in detail in the embodiments of this disclosure. The intermediate representation can be an intermediate language description between the programming language and the machine-readable instruction sequence, and can also be called a computation graph. The computation graph describes the computational logic of the neural network model through operator nodes and the dependencies between operator nodes. Operator nodes can include loading nodes, computing nodes, and storage nodes. The loading nodes can be used to load the input tensors required by the computing nodes for calculation. The computing nodes can be used to perform corresponding calculations on the input tensors. The storage nodes can be used to store the calculation results of the computing nodes. The computational graph of a neural network model is constructed through the dependency topology of multiple operator nodes. The types of operator nodes can correspond to the types of operations in the neural network model, such as operator nodes for convolution operations, operator nodes for pooling operations, and so on. The computational graph can also be divided into multiple computational subgraphs based on the different network layers in the neural network model. The specific configuration can be determined based on actual needs.

[0040] In an optional example, step 201 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first acquisition module executed by the processor.

[0041] In step 202, any network layer among at least one network layer containing a preset operation in the first model is used as a target network layer, and the preset operation of the target network layer is converted into a convolution operation supported by the neural network processor to obtain a converted first network layer, where the preset operation is an operation including a multiplication-accumulation operation.

[0042] Among them, the preset operations may include dot product operations, linear operations, matrix multiplication operations, and other operations including multiplication and accumulation operations. The dot product operation may refer to the sum of the products of corresponding elements of two vectors (one-dimensional tensors), such as vector a = (a1, a2, a3), vector b = (b1, b2, b3), and the dot product operation of vector a and vector b may be expressed as a·b = a1*b1+a2*b2+a3*b3. The matrix multiplication operation may represent the multiplication of an M*K matrix (two-dimensional tensor) A and a K*N matrix B to obtain an M*N result. It can be expressed as follows:

[0043] Linear operations can represent operations where a set of data is weighted according to certain weights, which can be expressed as follows: y = a1x1 + a2x2 + ...a m x m

[0044] Among them, a i (i=1,2,…,m, m is an integer greater than 1) represents x i The corresponding weight.

[0045] The data structure of the input tensor or output tensor of a network layer or operation in a neural network can include one-dimensional, two-dimensional, three-dimensional, four-dimensional, etc.

[0046] A tensor for a two-dimensional data structure can be called two-dimensional feature data, which can be expressed as a data plane with a certain height (H) and width (W) composed of multiple points. Any point in the data plane can be called a data item (or element) of the feature data, and can be represented by a two-tuple (h, w), where h can indicate which row the data item is located in the data plane from top to bottom (i.e., the positioning in the height dimension), and w can indicate which column the data item is located in the data plane from left to right (i.e., the positioning in the width dimension). The above two-tuple (h, w) can be called a data item position of the feature data, which is a two-dimensional positioning in the two dimensions of height and width. A data item with a data item position of (h, w) can correspond to a data item with row number h and column number w in the feature data. A tensor for a two-dimensional data structure can include, for example, input tensors and output tensors for matrix multiplication operations. A one-dimensional input tensor can be represented as two-dimensional feature data with a height or width of 1. For example, the input tensor of a dot product operation or a linear operation can be represented as a 1*K two-dimensional tensor, where 1 represents the height and K represents the width.

[0047] A tensor of a three-dimensional data structure can be called three-dimensional feature data, and can be expressed as a data cube composed of multiple points with a certain height (H), width (W) and depth (C). Any point in the data cube can be called a data item of feature data, and can be represented by a triple (h, w, c), where h can represent which row of the data cube the data item is located in from top to bottom (i.e., positioning in the height dimension), w can represent which column of the data cube the data item is located in from left to right (i.e., positioning in the width dimension), and c can represent which channel of the data cube the data item is located in from front to back (i.e., positioning in the depth dimension). A tensor of a three-dimensional data structure can, for example, include the input tensor of a convolutional layer or a convolution operation.

[0048] The tuple (h, w) in the above triple (h, w, c) can be called a data item position in the feature data, which is a two-dimensional location in the height and width dimensions. The data item at the data item position (h, w) can correspond to the data item in any channel with row number h and column number w in the feature data. Therefore, any data item in the feature data can be located by the corresponding data item position and the channel number of the channel.

[0049] For batched network layers or operations, in addition to the height and width dimensions, a count dimension can also be included, such as the input tensor for a batch matrix multiplication (Batch Matrix Multiplication). Batch matrix multiplication can be used to perform matrix multiplication of multiple left and right matrices, for example, by multiplying a batch of B*M*K matrices (i.e., B left matrices) with a corresponding batch of B*K*N matrices (i.e., B right matrices), resulting in a batch of B*M*N matrices. In this case, a three-dimensional tensor can be represented as a certain number of data planes with a certain height and width. Each data item can be represented as a triple (k, h, w), where k indicates the data plane in which the data item is located (i.e., its position in the count dimension), h indicates the row in which the data item is located in data plane k (i.e., its position in the height dimension), and w indicates the column in which the data item is located in data plane k (i.e., its position in the width dimension).

[0050] A tensor of a four-dimensional data structure can be called four-dimensional feature data, and can be expressed as a certain number of data cubes composed of multiple points, each data cube having a certain height, width and depth. Each data item in the feature data can be represented by a four-tuple (k, h, w, c), where k indicates which data cube the data item is located in (i.e., the positioning in the dimension of quantity), h can indicate which row the data item is located in from top to bottom in the data cube k (i.e., the positioning in the dimension of height), w can indicate which column the data item is located in from left to right in the data cube k (i.e., the positioning in the dimension of width), and c can indicate which channel the data item is located in from front to back in the data cube k (i.e., the positioning in the dimension of depth). A tensor of a four-dimensional data structure can, for example, include a batched convolution layer or the input tensor of a convolution operation.

[0051] The weight parameter (weight tensor) of the convolution layer or convolution operation in the neural network can be a four-dimensional data or a four-dimensional tensor, and each kernel of the weight parameter can be a three-dimensional data. Similar to the representation of feature data, each kernel of the weight parameter can also be expressed as a data cube with a certain height, width and depth composed of multiple points, and any point in the data cube can be called an element of the weight parameter or an element of a kernel of the weight parameter. Similar to the feature data, a triple (h, w, c) can be used to locate the elements in a kernel of the weight parameter. In addition, a four-tuple (k, h, w, c) can also be used to locate an element in the weight parameter, where k can indicate which core of the weight parameter the element is located in (i.e., positioning in the dimension of quantity), h can indicate which row the element is located in from top to bottom in the core k (i.e., positioning in the dimension of height), w can indicate which column the element is located in from left to right in the core k (i.e., positioning in the dimension of width), and c can indicate which channel the element is located in from front to back in the core k (i.e., positioning in the dimension of depth).

[0052] The quadruple (k, h, w, c) used to represent an element in the weight parameter or the binary (h, w) in the triple (h, w, c) used to represent an element in the kernel k can be called an element position in the kernel k of the weight parameter, which is a two-dimensional location in the height and width dimensions. In the kernel k, the element at the element position (h, w) can correspond to the element on any channel with row number h and column number w in the kernel k. Therefore, any element in the kernel k can be located by the corresponding element position and the channel number of the channel in which it is located.

[0053] In some optional embodiments, the network layer in the first model that contains the preset operation can be determined according to a preset judgment rule. The preset judgment rule can be determined based on the relevant operation parameters of each network layer of the neural network model described in the first model. For example, if the first model is an intermediate representation model, the network layer in the first model can be represented as a computational subgraph in the computational graph of the model, and whether the preset operation is included can be determined based on the operation description information of the operator node of the computational subgraph. If the first model is a model described in a programming language, whether the preset operation is included can be determined based on the specific operation description information of the programming language.

[0054] In some optional embodiments, a corresponding flag may be pre-set for the network layer containing the preset operation to facilitate determining whether the network layer contains the preset operation. As long as it can be determined whether the preset operation is included, the specific preset judgment rules are not limited in this embodiment.

[0055] In some optional embodiments, the convolution operation supported by the neural network processor may be a pointwise convolution operation. A pointwise convolution operation may not change the height and width dimensions of the input tensor, but only the number of channels of the input tensor. The convolution kernel size of a pointwise convolution may be 1*1, and the number of channels may be C. That is, each convolution kernel may be a vector containing C elements in the channel direction. When convolving an input tensor of H*W*C (height*width*number of channels), the convolution kernel can perform multiplication-accumulation operations with the elements of the C channels at each hw (h=1, 2, ..., H, w=1, 2, ..., W) position of the input tensor to obtain the convolution result of one channel corresponding to the convolution kernel. Each convolution kernel can obtain the convolution result of one channel. By setting the number of convolution kernels, the number of channels of the input tensor can be changed. For example, if the number of convolution kernels is C1, the final convolution result can be an output tensor of H*W*C1, where H represents the height of the output tensor, W represents the width of the output tensor, and C1 represents the number of channels of the output tensor. It can be seen that the point-by-point convolution process can implement multiplication-accumulation operations. Based on this, preset operations such as dot product operations, matrix multiplication operations, and linear operations including multiplication-accumulation operations can be converted into point-by-point convolution operations, so that the preset operations can be implemented through point-by-point convolution operations, so that the preset operations can be accelerated by the neural network processor and the model inference speed can be improved.

[0056] In some optional embodiments, at least one network layer may be configured as one or more network layers including preset operations based on actual needs. For example, at least one network layer may include all network layers including preset operations, without specific limitation. Each network layer in the at least one network layer including the preset operations may serve as a target network layer, and the preset operations in the target network layer may be converted into convolution operations supported by the neural network processor, so that the converted first network layer may be determined based on the converted convolution operations.

[0057] In some optional embodiments, converting the preset operation into a convolution operation can be achieved by reconstructing the computational subgraph of the target network layer. The operator node representing the preset operation in the computational subgraph can be converted into an operator node including an input tensor data structure transformation and a convolution operation according to certain conversion rules, so that the input tensor of the preset operation can adapt to the convolution operation, and then the output tensor corresponding to the preset operation can be obtained through the convolution operation. The converted operation corresponding to the preset operation and other operations in the target network layer other than the preset operation can be combined to obtain the converted first network layer. Based on this, the first network layer corresponding to each network layer containing the preset operation can be obtained.

[0058] In an optional example, step 202 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first processing module executed by the processor.

[0059] Step 203: Based on the converted first network layer, an optimized model corresponding to the first model is obtained.

[0060] After the optimization of all network layers that need to be converted is completed, an optimized model corresponding to the first model can be obtained based on the converted first network layer and other network layers except the network layer that needs to be converted.

[0061] In some optional embodiments, during the model optimization process, in addition to the optimization of converting the above-mentioned preset operation into a convolution operation, any other feasible optimization can be performed on the first model according to actual needs. Taking the first model of the intermediate representation as an example, for example, in the intermediate representation stage, the optimization of improving the utilization rate of the neural network processor by splitting the computational subgraph of the preset network layer, etc., is not limited in this embodiment. Ultimately, an optimized model can be obtained based on all optimized network layers. The optimized model can also be called a reconstructed optimized computational graph.

[0062] In an optional example, step 203 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by a second processing module executed by the processor.

[0063] Step 204: Generate a target instruction sequence executable by a neural network processor corresponding to the neural network model to be compiled based on the optimization model.

[0064] After obtaining the optimized model, the optimized model can be compiled into a target instruction sequence executable by the neural network processor through the relevant components and tool chain of the neural network compiler. For example, if the first model is a programming language model, the optimized model can be compiled into a target instruction sequence through the front-end, middle-end and back-end of the neural network compiler. For another example, if the first model is an intermediate representation model, the optimized model can be compiled into a target instruction sequence through the back-end components and tool chain of the neural network compiler.

[0065] In an optional example, step 204 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a third processing module executed by the processor.

[0066] The compilation method of the neural network model provided in this embodiment converts preset operations including dot product operations, matrix multiplication operations, linear operations, and multiplication-accumulation operations into convolution operations supported by the neural network processor during the model compilation process, so that the generated target instruction sequence can implement the preset operations through convolution operations, thereby enabling the above-mentioned preset operations of the model to be accelerated by the neural network processor used to accelerate the convolutional neural network, which helps to improve the model reasoning speed.

[0067] FIG3 is a flowchart of a method for compiling a neural network model provided by another exemplary embodiment of the present disclosure.

[0068] In some optional embodiments, after obtaining the first model corresponding to the neural network model to be compiled in step 201, the method of the embodiment of the present disclosure may further include:

[0069] Step 301: For any network layer of the first model, obtain the operating parameters of the network layer.

[0070] Among them, the operating parameters may be relevant parameters that describe the network layer. For example, the operating parameters may include network layer identification parameters, input-related parameters, operation-related parameters, output-related parameters, etc., which can be set according to actual needs. The operation-related information included in the operating parameters can be used to determine whether the network layer contains preset operations. For example, conv, pointwiseconv, etc. represent parameters of various convolution operations (which may include convolution kernel size, sliding step size, etc.), parameters representing matrix multiplication (which may include the first matrix, the second matrix, and the matrix multiplication operation), and so on. Different types of network layers may include different operating parameters, which can be set according to actual needs.

[0071] In some optional embodiments, the operation parameter may be descriptive information related to the operator node of the computation subgraph of the network layer, which is used to identify the operation type corresponding to the operator node in the computation subgraph, and is not specifically limited.

[0072] In an optional example, step 301 may be executed by the processor calling a corresponding instruction stored in a memory, or may be executed by a parameter acquisition module executed by the processor.

[0073] Step 302: Determine the state of the network layer including the preset operation according to the operation parameters and the preset judgment rule.

[0074] Among them, the preset judgment rules can be set according to the description of the relevant information of the preset operation in the operation parameters. The preset judgment rules can be used to determine the state of any network layer containing the preset operation according to the operation parameters, and the state can include the inclusion state and the non-inclusion state.

[0075] In an optional example, step 302 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first determination module executed by the processor.

[0076] Step 303 : In response to the state of including the preset operation being the included state, determining that the network layer is a network layer including the preset operation.

[0077] In some optional embodiments, each network layer of the first model can be traversed. For the traversed network layer, the state of the network layer containing the preset operation can be determined based on the operating parameters of the network layer. If the state is the inclusion state, it can be determined that the network layer is a network layer containing the preset operation, and then the optimization processing of the embodiment of the present disclosure to convert the preset operation into the convolution operation supported by the neural network processor can be performed on the network layer to obtain the optimization result corresponding to the network layer. If the network layers of the first model are traversed, the optimization of the first model is completed, and the optimization model corresponding to the first model is obtained.

[0078] In an optional example, step 303 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by a second determination module executed by the processor.

[0079] This embodiment identifies the network layer containing the preset operation in the first model through the preset judgment rule, thereby providing an effective reference basis for determining the network layer that needs to be optimized.

[0080] FIG4 is a flowchart of a method for compiling a neural network model provided by yet another exemplary embodiment of the present disclosure.

[0081] In some optional embodiments, step 202 uses any one of the at least one network layers in the first model that includes a preset operation as a target network layer, converts the preset operation of the target network layer into a convolution operation supported by the neural network processor, and obtains the converted first network layer, which may specifically include the following steps:

[0082] Step 2021: Use any network layer in at least one network layer including a preset operation in the first model as a target network layer.

[0083] In an optional example, step 2021 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first determining unit executed by the processor.

[0084] Step 2022: Determine a first input tensor and a second input tensor corresponding to a preset operation in the target network layer.

[0085] Among them, the first input tensor and the second input tensor can be determined based on the input tensor of the target network layer and other operations before the preset operation in the target network layer. For example, if the preset operation is the first operation of the target network layer or the target network layer only includes the preset operation, the first input tensor and the second input tensor can be determined based on the input tensor of the target network layer. If other operations are included before the preset operation, and the input of the preset operation is the output of other operations, the first input tensor and the second input tensor can be determined based on the output tensor of other operations. Depending on the specific type of the preset operation, the method of determining the first input tensor and the second input tensor may be different. For example, for matrix multiplication operation, the preset operation includes two tensors, the left matrix (first matrix) and the right matrix (second matrix), then these two tensors can be used as the first input tensor and the second input tensor corresponding to the preset operation respectively. For linear operation, the input of the linear operation only includes one input tensor, and the linear transformation matrix used for the linear operation is the weight parameter obtained by training the target network layer. Then the input tensor of the linear operation can be used as the first input tensor, and the linear transformation matrix can be used as the second input tensor. Therefore, the first input tensor and the second input tensor may be input tensors corresponding to a preset operation determined according to certain rules, and may not be the original input tensors of the preset operation, so as to be used for conversion in subsequent operations.

[0086] In some optional embodiments, depending on the dimensions of the original input tensor of the preset operation, the number of the determined first input tensors may be one or more. Similarly, the number of the second input tensors may be one or more.

[0087] For example, for a batch matrix multiplication operation, the dimensions of the original input tensor can be a batch matrix of B*M*K (i.e., B left matrices) and a batch matrix of B*K*N (i.e., B right matrices). The number of first input tensors determined is B, and the number of second input tensors is B. The first input tensor corresponds one-to-one with the second input tensor and can be used for B matrix multiplication operations. Each matrix multiplication operation can be converted into a convolution operation. Therefore, the batch matrix multiplication operation can be converted into B convolution operations. For preset operations with original input tensors of higher dimensions, the first input tensor and the second input tensor can be determined based on the specific characteristics of the operation and the specific characteristics of the convolution operation. For example, the original input tensor is an I*J*M*T tensor and a K*T*N tensor, where I*J*M*T represents an M*T matrix array with I rows and J columns (including I*J left matrices with a height of M and a width of T), and K*T*N represents K T*N right matrices, each with a height of T and a width of N. I*J M*T left matrices are respectively subjected to matrix multiplication operations with K T*N right matrices to obtain I*J*K M*N result matrices. It is then determined that the number of first input tensors corresponding to the preset operation is I*J, and the number of second input tensors is K. Each first input tensor corresponds to each second input tensor for matrix multiplication operations, so that it can be determined that I*J*K matrix multiplication operations are required, and each matrix multiplication operation is converted into a corresponding convolution operation, so that the preset operation can be converted into I*J*K convolution operations.

[0088] In an optional example, step 2022 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a second determining unit executed by the processor.

[0089] Step 2023: Based on the first transformation rule, perform data structure transformation on the first input tensor to obtain a transformed third input tensor.

[0090] Among them, the first transformation rule can be set according to the characteristics of the preset operation and the characteristics of the convolution operation. The setting principle can be to enable the third input tensor obtained by the transformation to be used as the input tensor of the convolution operation, and to obtain the operation result of the preset operation through the convolution operation. The specific transformation rules are not limited.

[0091] Exemplarily, the preset operation takes matrix multiplication as an example. The first input tensor is the left matrix, which is a two-dimensional tensor. The input tensor of the convolution operation supported by the neural network processor is a four-dimensional tensor. Then, the two-dimensional matrix can be converted into a four-dimensional tensor suitable for the convolution operation. For example, the first input tensor of M*K (M represents the height of the first input tensor, and K represents the width of the first input tensor) is transformed into a third input tensor of 1*1*M*K. The first 1 in 1*1*M*K represents the number of three-dimensional feature data included in the third input tensor, the second 1 represents the height of the three-dimensional feature data of the third input tensor, M represents the width of the three-dimensional feature data of the third input tensor, and K represents the number of channels of the three-dimensional feature data of the third input tensor.

[0092] In an optional example, step 2023 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first processing unit executed by the processor.

[0093] Step 2024: Based on the second transformation rule, perform data structure transformation on the second input tensor to obtain a transformed fourth input tensor.

[0094] Among them, the second transformation rule can be set according to the characteristics of the preset operation and the convolution operation. The setting principle can be to enable the fourth input tensor obtained by the transformation to be used as the weight tensor of the convolution operation (or called the convolution weight tensor), and to obtain the operation result of the preset operation through the convolution operation. The specific transformation rules are not limited.

[0095] Exemplarily, the preset operation takes matrix multiplication operation as an example. The second input tensor corresponding to the preset operation is the right matrix, such as a K*N matrix. The convolution weight tensor of the point-by-point convolution operation includes one or more convolution kernels. The dimension of each convolution kernel is 1*1*C three-dimensional data. The first 1 represents the height of the convolution kernel, the second 1 represents the width of the convolution kernel, and C represents the number of channels of the convolution kernel. Then the convolution weight tensor can be a four-dimensional tensor of N*1*1*C, where N represents the number of convolution kernels, that is, N 1*1*C three-dimensional convolution kernels. Based on this, combined with the specific operation of the convolution operation and the relevant characteristics of the matrix multiplication operation, the two-dimensional second input tensor can be transformed into a fourth input tensor, so that the fourth input tensor can be used as a weight tensor to convolve the above-mentioned third input tensor to obtain the output tensor corresponding to the preset operation. For example, the number of channels C of the convolution weight tensor is the number of rows K of the second input tensor, and the number of convolution kernels of the convolution weight tensor is the number of columns N of the second input tensor.

[0096] Step 2023 and step 2024 are performed in no particular order.

[0097] In an optional example, step 2024 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a second processing unit executed by the processor.

[0098] Step 2025: Use the third input tensor as the input tensor of the convolution operation, and use the fourth input tensor as the convolution weight tensor of the convolution operation.

[0099] Among them, the convolution weight tensor may include one or more convolution kernels for performing convolution operations on the third input tensor, and the preset multiplication and accumulation operations can be implemented through the convolution operation of each convolution kernel, and then the preset operations such as matrix multiplication operations, dot product operations, and linear operations can be implemented through one or more convolution kernels.

[0100] In an optional example, step 2025 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a third determining unit executed by the processor.

[0101] Step 2026: Perform a convolution operation based on the third input tensor and the convolution weight tensor to obtain a first output tensor.

[0102] Among them, the convolution operation can be a point-by-point convolution operation, and the third input tensor can be weighted in the depth direction (channel direction) by one or more convolution kernels included in the convolution weight tensor to obtain a first output tensor with the same number of channels as the number of convolution kernels. The dimension of the first output tensor can be H*W*N, where H represents height, W represents width, and N represents the number of channels. N can also represent the number of convolution kernels in the convolution weight tensor. Since point-by-point convolution does not change the size of the input tensor in the height and width dimensions but only changes the number of channels, H*W is the same size as the third input tensor.

[0103] In some optional embodiments, the number of first input tensors can be multiple. For example, if matrix multiplication operations of multiple left matrices with the same right matrix are required, the number of first input tensors is multiple, and each first input tensor can be transformed according to the above transformation rules. Alternatively, multiple first input tensors can be transformed together as a batch, and the multiple first input tensors can be represented as an S*M*K data structure, which is transformed into a third input tensor of S*1*M*K, and convolved with the N*1*1*K convolution weight tensor to obtain S 1*M*N first output tensors.

[0104] In some optional embodiments, the number of the first input tensors and the number of the second input tensors can both be multiple, and the number of the first input tensors and the number of the second input tensors can be the same. For example, in a batch matrix multiplication operation, matrix multiplication of a batch of B*M*K matrices (i.e., B left matrices) can be performed with a corresponding batch of B*K*N matrices (i.e., B right matrices) to obtain a batch of B*M*N matrices. The batch matrix multiplication can then be split into B matrix multiplications and processed according to the above-mentioned processing flow of the matrix multiplication operation.

[0105] In some optional embodiments, the number of first input tensors and the number of second input tensors are both multiple, and the number of first input tensors and the number of second input tensors may be different. For example, in the above example, the original input tensors are I*J*M*T tensors and K*T*N tensors, and the I*J M*T left matrices determined are used as I*J first input tensors, and the K T*N right matrices are used as K second input tensors, forming I*J*K input tensor pairs. Each input tensor pair may include a first input tensor and a second input tensor, corresponding to a matrix multiplication operation. The first input tensor and the second input tensor of each input tensor pair are converted according to the above steps to obtain the corresponding third input tensor and fourth input tensor, and then obtain I*J*K first output tensors.

[0106] In practical applications, when converting a predefined operation into multiple convolution operations, these operations can be executed in parallel or serially, depending on the scheduling rules of the neural network processor. For example, if the neural network processor has sufficient resources, the computing resources of multiple convolution operations can be called upon to execute multiple convolution operations in parallel.

[0107] In an optional example, step 2026 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a third processing unit executed by the processor.

[0108] Step 2027: Determine a second output tensor of the preset operation based on the first output tensor.

[0109] Among them, the second output tensor can be obtained by performing a data structure transformation on the first output tensor. Since the first output tensor obtained by the convolution operation is usually a three-dimensional or four-dimensional tensor, the output tensor of the preset operation can be a one-dimensional or two-dimensional tensor, or the actual data items located in each dimension of the data structure of the output tensor of the preset operation may be different from the first output tensor of the convolution operation. For example, the number of channels of the convolution result may correspond to the width of the output result of the preset operation, and the width of the convolution result may correspond to the height of the output result of the preset operation. The first output tensor is transformed into a tensor with the same data structure as the output tensor of the preset operation through data structure transformation. The specific transformation of the first output tensor to the second output tensor can be determined according to the specific preset operation, and the embodiments of the present disclosure are not limited thereto.

[0110] For example, for the matrix multiplication operation of two matrices of M*K (i.e. height*width) and K*N (i.e. height*width), the result of the matrix multiplication is a matrix of M*N (i.e. height*width), and the first output tensor obtained by the convolution operation is a tensor of 1*M*N (i.e. height*width*number of channels). The result matrix of the matrix multiplication operation of M*N (i.e. height*width) can be obtained through data structure transformation.

[0111] In an optional example, step 2027 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the fourth processing unit executed by the processor.

[0112] Step 2028: Based on the second output tensor, determine the target output tensor of the first network layer to obtain the first network layer.

[0113] Among them, if the preset operation is the last operation or the only operation of the target network layer, the second output tensor can be used as the output tensor of the target network layer, and is also the target output tensor of the optimized first network layer. After the above transformation, the target network layer can be transformed into the first network layer based on the convolution operation.

[0114] In an optional example, step 2028 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the fifth processing unit executed by the processor.

[0115] This embodiment determines the first input tensor and the second input tensor of the preset operation, and then performs data structure transformation on the first input tensor and the second input tensor according to different transformation rules, so that the obtained third input tensor and fourth input tensor can be used as the input tensor and convolution weight tensor of the convolution operation, respectively, to facilitate obtaining the operation result of the preset operation through the convolution operation, thereby laying the foundation for realizing acceleration of the preset operation based on a neural network processor that accelerates the convolution operation, and helping to further improve the model inference speed.

[0116] In some optional embodiments, determining the first input tensor and the second input tensor corresponding to the preset operation in the target network layer in step 2022 may include:

[0117] Determine the operation type of the preset operation; in response to the operation type of the preset operation being a dot product operation, use the first tensor and the second tensor to be subjected to the dot product operation as the first input tensor and the second input tensor respectively; or, in response to the operation type of the preset operation being a matrix multiplication operation, use the first matrix and the second matrix to be subjected to the matrix multiplication operation as the first input tensor and the second input tensor respectively; or, in response to the operation type of the preset operation being a linear operation, use the original input tensor of the linear operation as the first input tensor, and use the linear transformation matrix of the linear operation as the second input tensor.

[0118] Among them, the first tensor and the second tensor can be one-dimensional tensors, that is, vectors. The first matrix and the second matrix can be two-dimensional tensors. Matrix multiplication operations can include two-dimensional matrix multiplication operations and multi-dimensional matrix multiplication operations. Depending on the dimension of the matrix multiplication operation, the number of first matrices and the number of second matrices can be different. For example, the multi-dimensional matrix multiplication operation of the above-mentioned batch matrix multiplication operation, the multi-dimensional matrix multiplication operation of the tensor of I*J*M*T and the tensor of K*T*N, etc. The dot product operation can also be called a one-dimensional matrix multiplication operation. For specific dot product operations, matrix multiplication operations and linear operations, please refer to the above content and will not be repeated here. The linear transformation matrix can be a weight parameter obtained through model training. The linear operation is, for example, a fully connected layer in the model. Each network node in the current layer is obtained by performing a linear operation (weighted) on the output of each node in the previous layer according to the corresponding weight. The dot product operation can be, for example, a self-attention network layer in the model, which can calculate the dot product of the query vector and the key vector. The matrix multiplication operation can be, for example, the self-attention network layer in the model, which can perform matrix multiplication on the input tensor and the mapping tensor (which can include the query mapping tensor, the key mapping tensor, and the value mapping tensor) to obtain the query tensor, the key tensor, and the value tensor.

[0119] This embodiment determines the first input tensor and the second input tensor corresponding to the preset operation for different operation types, and can convert matrix multiplication operations, dot product operations and linear operations into point-by-point convolution operations supported by the neural network processor, so that the matrix multiplication operations, dot product operations and linear operations in the model can be accelerated based on the neural network processor, which helps to improve the model reasoning speed.

[0120] In some optional embodiments, step 2023 of performing data structure transformation on the first input tensor based on the first transformation rule to obtain a transformed third input tensor may include:

[0121] The data structure of the first input tensor is transformed from M*K to 1*1*M*K to obtain a third input tensor, where M represents the number of rows of the first input tensor and K represents the number of columns of the first input tensor.

[0122] In which, when M is greater than 1, the first input tensor can represent the first matrix of the matrix multiplication operation; when M is equal to 1 and K is greater than 1, the first input tensor can represent the first tensor of the dot product operation or the input tensor of the linear operation.

[0123] In an optional example, FIG5 is a schematic diagram of the data structure transformation principle of the first input tensor provided by an exemplary embodiment of the present disclosure. As shown in FIG5 , M=4, K=6, that is, the first input tensor is a two-dimensional tensor of 4 rows and 6 columns, and aij (i=1, 2, ..., M, j=1, 2, ..., K) in the gray box represents the element of the i-th row and j-th column of the first input tensor. The third input tensor obtained by the transformation is a four-dimensional tensor of 1*H1*W1*C1, where 1 in 1*H1*W1*C1 represents Batch, that is, the number of the first input tensors of M*K, H1 represents the height of the third input tensor, H1=1, W1 represents the width of the third input tensor, W1=M=4, C1 represents the number of channels of the third input tensor, C1=K=6. It can be seen that the 6 channels at each position in the third input tensor include the 6 elements of the corresponding row of the first input tensor. For example, position 11 includes the 6 elements of a11, a12, ..., a16 in the first input tensor along the channel direction, which can be multiplied and accumulated with the convolution kernel of each 6 channels of the convolution weight tensor during convolution.

[0124] This embodiment transforms the data structure of the first input tensor from M*K to 1*1*M*K so that the number of channels of the transformed third input tensor is K. During the convolution operation, the K elements of each row of the first input tensor and the K elements of the convolution kernel formed by each column of the second input tensor are multiplied and accumulated to obtain the multiplication and accumulation results of each row of the first input tensor and each column of the second input tensor. The multiplication and accumulation results are an element of the result matrix of the matrix multiplication of the first input tensor and the second input tensor, thereby preparing for the preset operation based on the convolution operation.

[0125] In some optional embodiments, performing data structure transformation on the second input tensor based on the second transformation rule to obtain a transformed fourth input tensor in step 2024 may include:

[0126] The data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain a fourth input tensor, where K in K*N represents the number of rows (i.e., height) of the second input tensor, and N in K*N represents the number of columns (i.e., width) of the second input tensor. N in N*1*1*K represents the amount of three-dimensional data included in the fourth input tensor, 1*1 represents the height*width of the fourth input tensor, and K represents the number of channels of the fourth input tensor.

[0127] In which, when N is greater than 1, the second input tensor can represent the second matrix of the matrix multiplication operation; when N=1 and K is greater than 1, the second input tensor can represent the second tensor of the dot product operation or the linear transformation matrix of the linear operation.

[0128] In an optional example, FIG6 is a schematic diagram of the data structure transformation principle of the second input tensor provided by an exemplary embodiment of the present disclosure. Corresponding to the first input tensor of FIG5 above, as shown in FIG6, the second input tensor is a two-dimensional tensor with a row number K of 6 and a column number N of 3, and bjn (j = 1, 2, ..., K, n = 1, 2, ..., N) represents the element of the jth row and nth column of the second input tensor. The fourth input tensor obtained by the data structure transformation is a four-dimensional tensor of N*H2*W2*C2, where N represents the number of batches of the fourth input tensor, H2 represents the height of the fourth input tensor, W2 represents the width of the fourth input tensor, and C2 represents the number of channels of the fourth input tensor. It can be seen that the second input tensor can be transformed into a fourth input tensor including 3 1*1*6 vectors, and each 1*1*6 vector can be used as a convolution kernel to perform a convolution operation with the above-mentioned third input tensor.

[0129] This embodiment transforms the second input tensor from K*N to N*1*1*K, so that the obtained fourth input tensor can be used as N 1*1*K convolution kernels to perform convolution operation with the above-mentioned third input tensor. The convolution kernel includes K elements of a column of the second input tensor in the channel direction. When performing the convolution operation, it can be multiplied and accumulated with the elements of the K channels at each position of the third input tensor to realize the multiplication and accumulation of each row of the first input tensor and each column of the second input tensor. Based on this, the operation result of the preset operation can be obtained through the convolution operation, which facilitates the acceleration of the preset operation through the neural network processor and improves the model inference speed.

[0130] In some optional embodiments, transforming the data structure of the second input tensor from K*N to N*1*1*K to obtain the fourth input tensor may include:

[0131] The data structure of the second input tensor is transformed from K*N to K*1*1*N to obtain a first intermediate tensor; the first intermediate tensor is used as a weight to perform a convolution operation on the preset unit tensor to obtain a fourth input tensor of N*1*1*K.

[0132] Among them, the preset unit tensor can be a 1*1*N*N unit tensor obtained by performing data structure transformation based on the N*N unit matrix.

[0133] In an optional example, Figure 7 is a schematic diagram of the data structure transformation principle of the second input tensor provided by another exemplary embodiment of the present disclosure. As shown in Figure 7, the two-dimensional second input tensor of K*N (i.e., height*width) can be transformed into a first intermediate tensor of K*1*1*N (i.e., quantity*height*width*number of channels), as K 1*1*N convolution kernels, and a Pointwise convolution operation is performed on a preset unit tensor of 1*1*N*N (i.e., quantity*height*width*number of channels) corresponding to the unit matrix of N*N (i.e., height*width), to obtain a fourth input tensor of N*1*1*K (i.e., quantity*height*width*number of channels).

[0134] It should be noted that the transformation of the tensor data structure can only change the arrangement rules of each element in the tensor without changing the specific element values, so that the transformed data structure can adapt to the data structure requirements of the corresponding operations of the neural network model, so that the corresponding data can be accurately loaded in the neural network processor to complete the model reasoning.

[0135] In an optional example, FIG8 is a schematic diagram of the convolution operation principle of the third input tensor and the convolution weight tensor provided by an exemplary embodiment of the present disclosure. As shown in FIG8, the convolution weight tensor may include N 1*1*K (i.e., height*width*number of channels) convolution kernels, and the convolution weight tensor may be point-by-point convolved with the third input tensor to obtain a first output tensor of 1*M*N, where 1 represents the height of the first output tensor, M represents the width of the first output tensor, and N represents the number of channels of the first output tensor. Based on the first output tensor, a second output tensor of M*N (height*width) may be obtained, which is the operation result corresponding to the preset operation. cin (i=1, 2, ..., M, n=1, 2, ..., N) represents the element of the i-th row and n-th column of the second output tensor. Taking c11 as an example (other cins are similar), c11=a11*b11+a12*b21+...+a1K*bK1. It can be seen that the second output tensor is the result of the matrix multiplication operation of the first input tensor and the second input tensor. When M=N=1, the second output tensor may be a result of a vector dot product operation or a linear operation, and details are omitted.

[0136] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0137] Any of the neural network model compilation methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the neural network model compilation methods provided in the embodiments of the present disclosure can be executed by a processor, such as a processor executing any of the neural network model compilation methods mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0138] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0139] FIG9 is a flowchart of a method for reasoning a neural network model according to an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as an in-vehicle computing platform. As shown in FIG9 , the apparatus according to the embodiment of the present disclosure may include the following steps:

[0140] Step 401: Obtain the input tensor corresponding to the neural network model and the target instruction sequence corresponding to the neural network model.

[0141] Among them, the target instruction sequence can be obtained based on the compilation method of the neural network model provided in any of the above embodiments.

[0142] In an optional example, step 401 may be executed by the processor calling a corresponding instruction stored in a memory, or may be executed by a second acquisition module executed by the processor.

[0143] Step 402: transmit the input tensor and the target instruction sequence to the neural network processor.

[0144] Among them, the neural network processor may include a computing array for performing various neural network operations. The computing array may include multiple rows and columns of computing units. The computing units may include, for example, multipliers, adders, multiplier-accumulators, etc., which can be specifically set according to actual needs.

[0145] In some optional embodiments, the input tensor and target instruction sequence may be transmitted by writing the input tensor and target instruction sequence into a storage unit in the neural network processor or a storage device external to the neural network processor, and transmitting the storage address to the neural network processor, which then reads the input tensor and target instruction sequence from the corresponding storage area. The specific method of calling the neural network processor is not limited.

[0146] In some optional embodiments, the number of neural network processors can be one or more. For a target instruction sequence corresponding to a neural network model, multiple neural network processors can be called to implement model reasoning based on the different network layers or different computing functions of the model, thereby improving processing efficiency or increasing the utilization of the neural network processors. For example, when multiple or a large number of images require target detection, certain scheduling rules can be set to schedule multiple neural network accelerators to work together based on rules such as load balancing or improving utilization of multiple neural network processors. The specific settings can be based on actual needs.

[0147] In an optional example, step 402 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a transmission module executed by the processor.

[0148] Step 403: Process the input tensor based on the neural network processor executing the target instruction sequence to obtain an inference result corresponding to the input tensor.

[0149] Among them, the neural network processor can execute the target instruction sequence. According to the optimized instruction sequence, during the inference process, the neural network processor can implement the original preset operation of the neural network model based on the convolution operation, obtain the operation result corresponding to the preset operation, and combine it with other inference processes to obtain the final inference result, further improving the model inference speed. In the corresponding application scenarios, it can improve the real-time performance of the application scenarios. For example, in the autonomous driving scenario, it can improve the speed of image processing such as target detection and semantic segmentation. For the perception of the surrounding environment of autonomous driving, it can provide corresponding perception results in more real time, allowing the vehicle to detect surrounding abnormalities more promptly, so as to carry out corresponding regulations and controls as early as possible, and improve the safety of vehicle driving. Or in the smart cockpit scenario, it can improve the real-time performance of the cockpit user's perception results to respond to user interactions more promptly and improve the user experience.

[0150] In an optional example, step 403 may be executed by a processor calling corresponding instructions stored in a memory, or may be executed by a neural network processor.

[0151] Any of the neural network model inference methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the neural network model inference methods provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any of the neural network model inference methods mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0152] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0153] Exemplary devices

[0154] Figure 10 is a schematic diagram of the structure of a neural network model compilation device provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding neural network model compilation method embodiment of the present disclosure. The device shown in Figure 10 may include: a first acquisition module 501, a first processing module 502, a second processing module 503, and a third processing module 504.

[0155] The first acquisition module 501 can be used to obtain a first model corresponding to the neural network model to be compiled.

[0156] The first processing module 502 can be used to take any network layer in at least one network layer in the first model that includes a preset operation as a target network layer, convert the preset operation of the target network layer into a convolution operation supported by the neural network processor, and obtain the converted first network layer, where the preset operation is an operation including a multiplication and accumulation operation.

[0157] The second processing module 503 may be configured to obtain an optimized model corresponding to the first model based on the converted first network layer.

[0158] The third processing module 504 may be configured to generate, based on the optimization model, a target instruction sequence executable by a neural network processor corresponding to the neural network model to be compiled.

[0159] FIG11 is a schematic structural diagram of a compilation device for a neural network model provided by another exemplary embodiment of the present disclosure.

[0160] In some optional embodiments, the apparatus of the embodiment of the present disclosure may further include: a parameter acquisition module 601 , a first determination module 602 , and a second determination module 603 .

[0161] The parameter acquisition module 601 can be used to acquire the operating parameters of any network layer of the first model.

[0162] The first determining module 602 may be configured to determine, based on operating parameters and preset judgment rules, whether the network layer includes a preset operation.

[0163] The second determining module 603 may be configured to determine that the network layer is a network layer including the preset operation in response to the state including the preset operation being the including state.

[0164] FIG12 is a schematic structural diagram of a first processing module 502 provided by an exemplary embodiment of the present disclosure.

[0165] In some optional embodiments, the first processing module 502 may include: a first determination unit 5021, a second determination unit 5022, a first processing unit 5023, a second processing unit 5024, a third determination unit 5025, a third processing unit 5026, a fourth processing unit 5027 and a fifth processing unit 5028.

[0166] The first determining unit 5021 may be configured to select any network layer in at least one network layer including a preset operation in the first model as a target network layer.

[0167] The second determining unit 5022 may be configured to determine a first input tensor and a second input tensor corresponding to a preset operation in a target network layer.

[0168] The first processing unit 5023 can be used to perform data structure transformation on the first input tensor based on the first transformation rule to obtain a transformed third input tensor.

[0169] The second processing unit 5024 can be used to perform data structure transformation on the second input tensor based on the second transformation rule to obtain a transformed fourth input tensor.

[0170] The third determining unit 5025 can be used to use the third input tensor as the input tensor of the convolution operation and the fourth input tensor as the convolution weight tensor of the convolution operation.

[0171] The third processing unit 5026 can be used to perform a convolution operation based on the third input tensor and the convolution weight tensor to obtain a first output tensor.

[0172] The fourth processing unit 5027 can be used to determine a second output tensor of a preset operation based on the first output tensor.

[0173] The fifth processing unit 5028 can be used to determine the target output tensor of the first network layer based on the second output tensor to obtain the first network layer.

[0174] In some optional embodiments, the second determining unit 5022 may be specifically configured to:

[0175] Determine the operation type of the preset operation; in response to the operation type of the preset operation being a dot product operation, use the first tensor and the second tensor to be subjected to the dot product operation as the first input tensor and the second input tensor respectively; or, in response to the operation type of the preset operation being a matrix multiplication operation, use the first matrix and the second matrix to be subjected to the matrix multiplication operation as the first input tensor and the second input tensor respectively; or, in response to the operation type of the preset operation being a linear operation, use the original input tensor of the linear operation as the first input tensor, and use the linear transformation matrix of the linear operation as the second input tensor.

[0176] In some optional embodiments, the first processing unit 5023 may be specifically configured to:

[0177] The data structure of the first input tensor is transformed from M*K to 1*1*M*K to obtain a third input tensor, where M represents the number of rows of the first input tensor and K represents the number of columns of the first input tensor.

[0178] In some optional embodiments, the second processing unit 5024 may be specifically configured to:

[0179] The data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain a fourth input tensor, where K represents the number of rows of the second input tensor and N represents the number of columns of the second input tensor.

[0180] In some optional embodiments, the second processing unit 5024 may be specifically configured to:

[0181] The data structure of the second input tensor is transformed from K*N to K*1*1*N to obtain a first intermediate tensor; the first intermediate tensor is used as a weight to perform a convolution operation on the preset unit tensor to obtain a fourth input tensor of N*1*1*K.

[0182] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0183] FIG13 is a schematic diagram of the structure of an inference device for a neural network model provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding inference method embodiment of the neural network model of the present disclosure. The device shown in FIG13 may include: a second acquisition module 701, a transmission module 702, and a neural network processor 703.

[0184] The second acquisition module 701 can be used to obtain the input tensor corresponding to the neural network model and the target instruction sequence corresponding to the neural network model.

[0185] Among them, the target instruction sequence can be obtained based on the compilation method of the neural network model provided in any of the above embodiments.

[0186] The transmission module 702 can be used to transmit the input tensor and the target instruction sequence to the neural network processor 703.

[0187] The neural network processor 703 can be used to execute a target instruction sequence to process the input tensor and obtain an inference result corresponding to the input tensor.

[0188] Among them, the number of neural network processors 703 can be one or more, and can be set according to actual needs.

[0189] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0190] Exemplary electronic devices

[0191] FIG14 is a structural diagram of an electronic device provided by an embodiment of the present disclosure, which includes at least one processor 11 and a memory 12 .

[0192] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0193] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.

[0194] In one example, the electronic device 10 may further include an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0195] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0196] The output device 14 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0197] Of course, for simplicity, FIG14 only shows some of the components of the electronic device 10 related to the present disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may further include any other appropriate components depending on the specific application.

[0198] Exemplary computer program products and computer-readable storage media

[0199] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the methods of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0200] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0201] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0202] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or component comprising electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0203] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be considered as essential to each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0204] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for compiling a neural network model, comprising: Obtain a first model corresponding to the neural network model to be compiled; Taking any network layer of at least one network layer in the first model that includes a preset operation as a target network layer, converting the preset operation of the target network layer into a convolution operation supported by a neural network processor, and obtaining a converted first network layer, wherein the preset operation is an operation including a multiplication-accumulation operation; Based on the converted first network layer, obtaining an optimization model corresponding to the first model; According to the optimization model, a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled is generated.

2. The method according to claim 1, wherein: The step of converting the preset operation of the target network layer into a convolution operation supported by a neural network processor to obtain a converted first network layer includes: Determine a first input tensor and a second input tensor corresponding to the preset operation; Based on a first transformation rule, performing a data structure transformation on the first input tensor to obtain a transformed third input tensor; Based on a second transformation rule, performing data structure transformation on the second input tensor to obtain a transformed fourth input tensor; Using the third input tensor as an input tensor for the convolution operation; Using the fourth input tensor as a convolution weight tensor for the convolution operation; Performing a convolution operation based on the third input tensor and the convolution weight tensor to obtain a first output tensor; Determining a second output tensor of the preset operation based on the first output tensor; Based on the second output tensor, a target output tensor of the first network layer is determined to obtain the first network layer.

3. The method according to claim 2, wherein: The step of performing data structure transformation on the first input tensor based on the first transformation rule to obtain a transformed third input tensor includes: The data structure of the first input tensor is transformed from M*K to 1*1*M*K to obtain the third input tensor, where M represents the number of rows of the first input tensor and K represents the number of columns of the first input tensor.

4. The method according to claim 2, wherein: The step of performing data structure transformation on the second input tensor based on the second transformation rule to obtain a transformed fourth input tensor includes: The data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain the fourth input tensor, wherein K represents the number of rows of the second input tensor and N represents the number of columns of the second input tensor.

5. The method according to claim 4, wherein: The step of transforming the data structure of the second input tensor from K*N to N*1*1*K to obtain the fourth input tensor includes: Transform the data structure of the second input tensor from K*N to K*1*1*N to obtain a first intermediate tensor; The first intermediate tensor is used as a weight, and a convolution operation is performed on the preset unit tensor to obtain the fourth input tensor of N*1*1*K.

6. The method according to claim 2, wherein: The determining the first input tensor and the second input tensor corresponding to the preset operation includes: Determining the operation type of the preset operation; In response to the operation type of the preset operation being a dot product operation, using the first tensor and the second tensor to be subjected to the dot product operation as the first input tensor and the second input tensor respectively; or, In response to the operation type of the preset operation being a matrix multiplication operation, using a first matrix and a second matrix to be subjected to a matrix multiplication operation as the first input tensor and the second input tensor respectively; or, In response to the operation type of the preset operation being a linear operation, the original input tensor of the linear operation is used as the first input tensor, and the linear transformation matrix of the linear operation is used as the second input tensor.

7. The method according to claim 1, after obtaining the first model corresponding to the neural network model to be compiled, further comprises: For any network layer of the first model, obtaining an operation parameter of the network layer; According to the operation parameters and the preset judgment rule, determining that the network layer includes the state of the preset operation; In response to the state being an inclusion state, the network layer is determined to be a network layer that includes the preset operation.

8. A method for reasoning a neural network model, comprising: Obtaining an input tensor corresponding to a neural network model and a target instruction sequence corresponding to the neural network model; The target instruction sequence is obtained based on the compilation method of the neural network model described in any one of claims 1 to 7; transmitting the input tensor and the target instruction sequence to a neural network processor; The input tensor is processed based on the neural network processor executing the target instruction sequence to obtain an inference result corresponding to the input tensor.

9. A compiling device for a neural network model, comprising: A first acquisition module, used to acquire a first model corresponding to the neural network model to be compiled; A first processing module, configured to use any network layer of at least one network layer containing a preset operation in the first model as a target network layer, convert the preset operation of the target network layer into a convolution operation supported by the neural network processor, and obtain a converted first network layer, wherein the preset operation is an operation including a multiplication-accumulation operation; A second processing module, configured to obtain an optimization model corresponding to the first model based on the converted first network layer; The third processing module is used to generate a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled according to the optimization model.

10. An inference device for a neural network model, comprising: A second acquisition module is used to acquire an input tensor corresponding to the neural network model and a target instruction sequence corresponding to the neural network model; The target instruction sequence is obtained based on the compilation method of the neural network model described in any one of claims 1 to 7; A transmission module, used for transmitting the input tensor and the target instruction sequence to a neural network processor; The neural network processor is used to execute the target instruction sequence to process the input tensor and obtain the inference result corresponding to the input tensor.

11. A computer-readable storage medium storing a computer program for executing the method for compiling a neural network model according to any one of claims 1 to 7.

12. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the compilation method of the neural network model described in any one of claims 1-7 above.

Citation Information

Patent Citations

  • Convolutional neural network operation acceleration method and device based on many-core processor

    CN111461311A

  • Data processing method, neural network training method and related device and apparatus

    CN113326930A

  • Neural network model compiling method and device, electronic equipment and storage medium

    CN114239803A

  • Neural network model compiling method and device, storage medium and electronic equipment

    CN115543337A

  • Neural network model compiling method and device, storage medium and electronic equipment

    CN115857940A

Cited By

  • Design method of MHCl binding peptide based on evolutionary information and Transform neural network algorithm

    CN120727089A

  • Method for designing mhc class ii binding peptides based on evolutionary information and transformer neural network algorithm

    CN120727089B