Compilation methods, inference methods, devices, equipment, and media of neural network models

Abstract

This disclosure provides a method for compiling a neural network model, an inference method, an apparatus, a device, and a medium. The compilation method includes: obtaining a first model corresponding to the neural network model to be compiled; selecting any one of at least one network layer in the first model containing preset operations as a target network layer; converting the preset operations of the target network layer into convolution operations supported by a neural network processor to obtain a converted first network layer, wherein the preset operations include multiplication-accumulation operations; obtaining an optimized model corresponding to the first model based on the converted first network layer; and generating a target instruction sequence executable by a neural network processor corresponding to the neural network model to be compiled based on the optimized model. This disclosure allows the implementation of preset operations including multiplication-accumulation operations through convolution operations, facilitating the acceleration of these preset operations by a neural network processor and significantly improving model inference speed.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for compiling, inference, apparatus, device, and medium for a neural network model. Background Technology

[0002] In recent years, Transformer-based neural network models (hereinafter referred to as "models") have been widely used in various fields or scenarios requiring image processing, text processing, and speech processing, such as visual perception scenarios for autonomous driving, medical image processing scenarios in the medical field, speech recognition scenarios for human-computer interaction, and office text recognition scenarios, etc. To improve the inference speed of neural network models, it is usually necessary to accelerate the inference process using a neural network processor. Neural network processors in related technologies are mainly used to accelerate Convolutional Neural Networks (CNNs). However, Transformer-based models typically also include operations such as dot product operations, matrix multiplication operations, and linear operations of linear layers. Directly supporting the acceleration of these operations results in a lower model inference speed. Summary of the Invention

[0003] To address the aforementioned technical issues such as low model inference speed, embodiments of this disclosure provide a method for compiling, inference, apparatus, device, and medium for neural network models. During model compilation, preset operations, including dot product, matrix multiplication, and linear operations (including multiply-accumulate operations), can be converted into convolution operations supported by a neural network processor. This allows the generated target instruction sequence executable by the neural network processor to perform the preset operations through convolution operations. Consequently, the preset operations of the model can be accelerated by a neural network processor used to accelerate convolutional neural networks, significantly improving model inference speed.

[0004] The first aspect of this disclosure provides a method for compiling a neural network model, comprising: obtaining a first model corresponding to the neural network model to be compiled; taking any one of at least one network layer containing a preset operation in the first model as a target network layer, converting the preset operation of the target network layer into a convolution operation supported by a neural network processor to obtain a converted first network layer, wherein the preset operation is an operation including multiplication-accumulation operation; obtaining an optimized model corresponding to the first model based on the converted first network layer; and generating a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled according to the optimized model.

[0005] A second aspect of this disclosure provides a reasoning method for a neural network model, comprising: obtaining an input tensor corresponding to the neural network model and a target instruction sequence corresponding to the neural network model; the target instruction sequence being obtained based on the compilation method of the neural network model described in any of the above embodiments; transmitting the input tensor and the target instruction sequence to a neural network processor; and processing the input tensor based on the target instruction sequence executed by the neural network processor to obtain a reasoning result corresponding to the input tensor.

[0006] A third aspect of this disclosure provides a neural network model compilation apparatus, comprising: a first acquisition module for acquiring a first model corresponding to a neural network model to be compiled; a first processing module for taking any one of at least one network layer containing a preset operation in the first model as a target network layer, converting the preset operation of the target network layer into a convolution operation supported by a neural network processor, thereby obtaining a converted first network layer, wherein the preset operation is an operation including multiplication-accumulation operations; a second processing module for obtaining an optimized model corresponding to the first model based on the converted first network layer; and a third processing module for generating a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled, based on the optimized model.

[0007] A fourth aspect of this disclosure provides an inference apparatus for a neural network model, comprising: a second acquisition module, configured to acquire an input tensor corresponding to the neural network model and a target instruction sequence corresponding to the neural network model; the target instruction sequence being obtained based on the compilation method of the neural network model described in any of the above embodiments; a transmission module, configured to transmit the input tensor and the target instruction sequence to a neural network processor; the neural network processor being configured to execute the target instruction sequence to process the input tensor and obtain an inference result corresponding to the input tensor.

[0008] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program for executing a compilation method of a neural network model according to any of the above embodiments of this disclosure, or for executing an inference method of a neural network model according to any of the above embodiments of this disclosure.

[0009] A sixth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement a method for compiling a neural network model according to any of the above embodiments of this disclosure.

[0010] A seventh aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, executes a compilation method for a neural network model provided in a first aspect embodiment of this disclosure, or executes an inference method for a neural network model provided in a second aspect embodiment of this disclosure.

[0011] Based on the neural network model compilation method, inference method, apparatus, device, and medium provided in the above embodiments of this disclosure, by converting preset operations such as dot product operation, matrix multiplication operation, linear operation, including multiplication-accumulation operation, into convolution operation supported by the neural network processor during the model compilation process, the generated target instruction sequence can realize the preset operations through convolution operation, thereby accelerating the above-mentioned preset operations of the model through the neural network processor used to accelerate the convolutional neural network, greatly improving the model inference speed. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a method for compiling a neural network model provided in an exemplary embodiment of this disclosure;

[0013] Figure 2 This is a flowchart illustrating a method for compiling a neural network model provided in an exemplary embodiment of this disclosure;

[0014] Figure 3 This is a flowchart illustrating a method for compiling a neural network model provided in another exemplary embodiment of this disclosure;

[0015] Figure 4 This is a flowchart illustrating a method for compiling a neural network model provided in yet another exemplary embodiment of this disclosure;

[0016] Figure 5 This is a schematic diagram illustrating the data structure transformation principle of the first input tensor provided in an exemplary embodiment of this disclosure;

[0017] Figure 6 This is a schematic diagram illustrating the data structure transformation principle of the second input tensor provided in an exemplary embodiment of this disclosure;

[0018] Figure 7 This is a schematic diagram illustrating the data structure transformation principle of the second input tensor provided in another exemplary embodiment of this disclosure;

[0019] Figure 8 This is a schematic diagram illustrating the convolution operation principle of the third input tensor and the convolution weight tensor provided in an exemplary embodiment of this disclosure;

[0020] Figure 9 This is a flowchart illustrating the inference method of a neural network model provided in an exemplary embodiment of this disclosure;

[0021] Figure 10 This is a schematic diagram of the structure of a compilation device for a neural network model provided in an exemplary embodiment of the present disclosure;

[0022] Figure 11 This is a schematic diagram of the structure of a compilation apparatus for a neural network model provided in another exemplary embodiment of this disclosure;

[0023] Figure 12 This is a schematic diagram of the structure of the first processing module 502 provided in an exemplary embodiment of this disclosure;

[0024] Figure 13 This is a schematic diagram of the structure of an inference device for a neural network model provided in an exemplary embodiment of the present disclosure;

[0025] Figure 14 This is a structural diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0027] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0028] This disclosure outlines

[0029] In the process of developing this disclosure, the inventors discovered that in recent years, Transformer-based neural network models (hereinafter referred to as "models") have been widely used in various fields or scenarios requiring image processing, text processing, and speech processing, such as visual perception scenarios for autonomous driving, medical image processing scenarios in the medical field, speech recognition scenarios for human-computer interaction, and office text recognition scenarios, etc. To improve the inference speed of neural network models, it is usually necessary to accelerate the inference process of the neural network model through a neural network processor. Neural network processors in related technologies are mainly used to accelerate the processing of Convolutional Neural Networks (CNNs). However, Transformer-based models typically also include operations such as dot product operations, matrix multiplication operations, and linear operations of linear layers. Neural network processors in related technologies often do not directly support the acceleration of these operations and require the participation of other processors, resulting in a lower model inference speed.

[0030] Exemplary Overview

[0031] Figure 1 This is an exemplary application scenario of the compilation method for the neural network model provided in this disclosure.

[0032] For neural network models with pre-defined operations such as dot product, matrix multiplication, and linear operations of linear layers, such as image processing models for image processing of the external environment or smart cockpit in autonomous driving scenarios, including object detection models, semantic segmentation models, and image classification models, and speech recognition models for in-vehicle user voice interaction, the compilation method of this disclosure can be used to take the corresponding neural network model as the neural network model to be compiled. By obtaining the first model corresponding to the neural network model to be compiled during the compilation stage, optimizing the first model, and converting at least one pre-defined operation contained in the first model into convolution operations supported by the neural network processor, an optimized model is obtained. Then, based on the optimized model, a target instruction sequence executable by the neural network processor is generated. The pre-defined operations can be implemented through convolution operations, thereby accelerating the above-mentioned pre-defined operations of the model through a neural network processor used to accelerate convolutional neural networks, greatly improving the model inference speed. The compilation of the neural network model can be achieved through a compiler, which can include a front-end, a middle-end, and a back-end. Each of the front-end, middle-end, and back-end includes different modular compiler components and toolchains. The front-end converts the neural network model to be compiled from a programming language into an intermediate representation model. The middle-end optimizes the intermediate representation model to obtain an optimized model. The back-end converts the optimized model into a machine-executable sequence of target instructions. The target instruction sequence includes a sequence of instructions executable by the neural network processor, which executes these instructions to complete real-time inference of the neural network. The neural network model compilation method disclosed herein can be implemented at any stage of the compilation process—front-end, middle-end, or back-end—without specific limitations. That is, the first model can be a model described by the programming language of the front-end, an intermediate representation model, or a model described by the initial instruction sequence generated by the back-end; the specific configuration can be determined according to actual needs.

[0033] The compilation method of the neural network model disclosed herein is applicable to accelerating scenarios or fields containing the above-mentioned preset operations on any neural network processor, such as autonomous driving scenarios, medical image processing scenarios, voice interaction scenarios, etc., and the specific scenarios and fields are not limited.

[0034] Exemplary methods

[0035] Figure 2 This is a flowchart illustrating a method for compiling a neural network model according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically servers, terminal devices, and other electronic equipment. Figure 2 As shown, it includes the following steps:

[0036] Step 201: Obtain the first model corresponding to the neural network model to be compiled.

[0037] The neural network model to be compiled can be a trained neural network model with at least one network layer containing preset operations, such as a trained Transformer-based object detection model, semantic segmentation model, etc. The specific type and function of the neural network model are not limited. The first model can be a model described in the programming language corresponding to the neural network model to be compiled, an intermediate representation model corresponding to the neural network model to be compiled, or a model described by the initial instruction sequence corresponding to the generated neural network model to be compiled. Taking the first model as an intermediate representation as an example, the neural network model to be compiled can be converted from a programming language to an intermediate representation using a neural network compiler to obtain the first model of the intermediate representation corresponding to the neural network model to be compiled. The specific method of converting from a programming language to an intermediate representation can be any feasible method according to actual needs, and this disclosure will not elaborate on it. The intermediate representation can be an intermediate language description between a programming language and a machine-recognizable instruction sequence, also known as a computation graph. The computation graph describes the computational logic of the neural network model through operator nodes and the dependencies between operator nodes. Operator nodes can include loading nodes, computation nodes, and storage nodes. Loading nodes are used to load the input tensors required for computation by the computation nodes, computation nodes are used to perform corresponding computations on the input tensors, and storage nodes are used to store the computation results of the computation nodes. The computation graph of a neural network model is constructed through the dependency topology of multiple operator nodes. The type of operator node corresponds to the type of operation in the neural network model; for example, it may include operator nodes for convolution operations, operator nodes for pooling operations, and so on. Depending on the different network layers in the neural network model, the computation graph can also be divided into multiple computation subgraphs, which can be set according to actual needs.

[0038] Step 202: Take any one of the network layers in the first model that contains preset operations as the target network layer, convert the preset operations of the target network layer into convolution operations supported by the neural network processor, and obtain the converted first network layer. The preset operations include multiplication and accumulation operations.

[0039] The preset operations can include dot product, linear operations, matrix multiplication, and multiplication-accumulation operations. Dot product refers to the sum of the products of corresponding elements of two vectors (one-dimensional tensors). For example, if vector a = (a1, a2, a3) and vector b = (b1, b2, b3), the dot product of vectors a and b can be expressed as a·b = a1*b1 + a2*b2 + a3*b3. Matrix multiplication represents the multiplication of an M*K matrix (two-dimensional tensor) A with a K*N matrix B to obtain an M*N result. This can be represented as follows:

[0040]

[0041]

[0042]

[0043] Linear operations represent weighted operations performed on a set of data according to certain weights, and can be represented as follows:

[0044] y = a1x1 + a2x2 + ... + a m x m

[0045] Among them, a i (i = 1, 2, ..., m, where m is an integer greater than 1) represents x i The corresponding weights.

[0046] The tensor data structure of the network layers or operations in a neural network can include one-dimensional, two-dimensional, three-dimensional, four-dimensional, and other types.

[0047] A two-dimensional data structure tensor can be called two-dimensional feature data. It can be represented as a data plane with a certain height (H) and width (W) composed of multiple points. Any point in this data plane can be called a data item (or element) of the feature data, and can be represented by a tuple (h, w). Here, h represents the row (height dimension) of the data item in the data plane from top to bottom, and w represents the column (width dimension) of the data item from left to right. This tuple (h, w) can be called the position of a data item in the feature data, representing its two-dimensional location in both height and width dimensions. A data item at position (h, w) corresponds to a data item in the feature data with row number h and column number w. Two-dimensional data structure tensors can include, for example, the input and output tensors of matrix multiplication operations. A one-dimensional input tensor can be represented as two-dimensional feature data with a height or width of 1. For example, the input tensor for dot product operations or linear operations can be represented as a 1*K two-dimensional tensor, where 1 represents the height and K represents the width.

[0048] A tensor in a three-dimensional data structure can be called three-dimensional feature data, and can be represented as a data cube composed of multiple points with a certain height (H), width (W), and depth (C). Any point in this data cube can be called a data item of the feature data, and can be represented by a triple (h, w, c), where h represents the row (i.e., height) of the data item in the data cube from top to bottom, w represents the column (i.e., width) of the data item in the data cube from left to right, and c represents the channel (i.e., depth) of the data item in the data cube from front to back. Tensors in three-dimensional data structures can include, for example, the input tensors of convolutional layers or convolution operations.

[0049] The pair (h, w) in the triplet (h, w, c) above can be referred to as a data item position in the feature data, which is a two-dimensional location in the dimensions of height and width. A data item at position (h, w) corresponds to a data item in any channel of the feature data with row number h and column number w. Therefore, any data item in the feature data can be located using its corresponding data item position and the channel number of its corresponding channel.

[0050] For batch processing network layers or operations, in addition to the dimensions of height and width, there is also a quantity dimension, such as the input tensor of batch matrix multiplication. Batch matrix multiplication is used to perform matrix multiplication of multiple left matrices and multiple right matrices respectively. For example, a batch of matrices of B*M*K (i.e., B left matrices) is multiplied by a batch of matrices of B*K*N (i.e., B right matrices) to obtain a batch of matrices of B*M*N. In this case, the tensor of the three-dimensional data structure is represented as a certain number of data planes with a certain height and width. Each data item can be represented as a triple (k, h, w), where k indicates which data plane the data item is located in (i.e., its location in the quantity dimension), h indicates which row in data plane k from top to bottom (i.e., its location in the height dimension), and w indicates which column in data plane k from left to right (i.e., its location in the width dimension).

[0051] A tensor in a four-dimensional data structure can be called four-dimensional feature data, and can be represented as a certain number of data cubes composed of multiple points, each data cube having a certain height, width, and depth. Each data item in this feature data can be represented by a quadruple (k, h, w, c), where k indicates which data cube the data item is located in (i.e., its location in the dimension of number), h indicates which row in data cube k from top to bottom (i.e., its location in the dimension of height), w indicates which column in data cube k from left to right (i.e., its location in the dimension of width), and c indicates which channel in data cube k from front to back (i.e., its location in the dimension of depth). Tensors in four-dimensional data structures can include, for example, the input tensors of batch convolutional layers or convolutional operations.

[0052] In neural networks, the weight parameters (weight tensors) of convolutional layers or convolution operations can be four-dimensional data or four-dimensional tensors, and each kernel of these weight parameters can be a three-dimensional data. Similar to the representation of feature data, each kernel of the weight parameters can also be represented as a data cube composed of multiple points with a certain height, width, and depth, and any point in this data cube can be called an element of the weight parameter or an element of a kernel of the weight parameter. Similar to feature data, a triple (h, w, c) can be used to locate elements within a kernel of the weight parameters. Alternatively, a quadruple (k, h, w, c) can be used to locate an element in the weight parameters, where k indicates which kernel of the weight parameters the element is located in (i.e., its location in the dimension of quantity), h indicates which row of kernel k in descending order (i.e., its location in the dimension of height), w indicates which column of kernel k in descending order (i.e., its location in the dimension of width), and c indicates which channel of kernel k in descending order (i.e., its location in the dimension of depth).

[0053] The tuple (h, w) in the quadruple (k, h, w, c) representing an element in the weight parameters or the triple (h, w, c) representing an element in the kernel k can be called an element position in the kernel k of the weight parameters. This position is a two-dimensional location in both height and width dimensions. In the kernel k, the element at position (h, w) corresponds to an element in any channel with row number h and column number w. Therefore, any element in the kernel k can be located using its corresponding element position and the channel number of its corresponding channel.

[0054] In some optional embodiments, the network layers containing preset operations in the first model can be determined according to preset judgment rules. These rules can be based on the relevant operation parameters described by each network layer of the neural network model in the first model. For example, if the first model is an intermediate representation, the network layers in the first model can be represented as computational subgraphs in the computation graph of the model. Whether or not a preset operation is included is determined based on the operation description information of the operator nodes in the computational subgraph. If the first model is a model described by a programming language, whether or not a preset operation is included can be determined based on the specific operation description information of the programming language.

[0055] In some optional embodiments, corresponding identifiers can be pre-set for network layers that contain preset operations to facilitate determining whether a network layer contains preset operations. As long as it is possible to determine whether a preset operation is contained, the specific preset judgment rules are not limited in this embodiment.

[0056] In some optional embodiments, the convolution operation supported by the neural network processor is pointwise convolution. Pointwise convolution does not change the height and width dimensions of the input tensor, but only changes the number of channels in the input tensor. The kernel size of pointwise convolution is 1*1, and the number of channels is C, that is, each convolution kernel is a vector containing C elements in the channel direction. When convolving an input tensor of H*W*C (height*width*number of channels), the convolution kernel is multiplied and accumulated with the elements of C channels at each position hw (h = 1, 2, ..., H, w = 1, 2, ..., W) of the input tensor to obtain the convolution result of one channel corresponding to the convolution kernel. Each convolution kernel yields a convolution result of one channel. By setting the number of convolution kernels, the number of channels in the input tensor is changed. For example, if the number of convolution kernels is C1, the final convolution result is an output tensor of H*W*C1, where H represents the height of the output tensor, W represents the width of the output tensor, and C1 represents the number of channels in the output tensor. As can be seen, the pointwise convolution process realizes the multiplication and accumulation operation. Based on this, the preset operations, including dot product operation, matrix multiplication operation, and linear operation, which include multiplication and accumulation operation, are converted into pointwise convolution operation. The preset operations are realized through pointwise convolution operation, which makes it easier to accelerate the preset operations through the neural network processor and improve the model inference speed.

[0057] In some optional embodiments, at least one network layer may be configured to include one or more network layers containing preset operations, depending on actual needs. For example, at least one network layer may include all network layers containing preset operations, without any specific limitation. Each network layer in the at least one network layer containing preset operations can be used as a target network layer, and its preset operations are converted into convolution operations supported by the neural network processor. The first network layer after conversion is determined based on the converted convolution operations.

[0058] In some optional embodiments, converting a preset operation into a convolution operation can be achieved by reconstructing the computation subgraph of the target network layer. The operator nodes representing the preset operation in the computation subgraph are converted into operator nodes that include input tensor data structure transformation and convolution operations according to certain transformation rules. This allows the input tensor of the preset operation to be adapted to the convolution operation, and then the output tensor corresponding to the preset operation is obtained through the convolution operation. Combining the converted operation corresponding to the preset operation with other operations in the target network layer besides the preset operation, the converted first network layer is obtained. Based on this, the first network layer corresponding to each network layer containing the preset operation can be obtained.

[0059] Step 203: Based on the transformed first network layer, obtain the optimized model corresponding to the first model.

[0060] Once the optimization of all network layers that need to be transformed is completed, the optimized model corresponding to the first model can be obtained based on the transformed first network layer and the other network layers besides the network layer that needs to be transformed.

[0061] In some optional embodiments, during model optimization, in addition to the aforementioned optimization of converting preset operations into convolution operations, other arbitrary feasible optimizations can be performed on the first model according to actual needs. Taking the intermediate representation of the first model as an example, for instance, optimizations could be made to improve the utilization of the neural network processor by segmenting the computational subgraph of the preset network layers during the intermediate representation stage, etc. This embodiment is not limited to these optimizations. Finally, an optimized model can be obtained based on all optimized network layers. The optimized model can also be referred to as the reconstructed optimized computational graph.

[0062] Step 204: Based on the optimization model, generate the target instruction sequence that the neural network processor can execute corresponding to the neural network model to be compiled.

[0063] After obtaining the optimized model, the neural network compiler's components and toolchain can be used to compile the optimized model into a sequence of target instructions executable by the neural network processor. For example, if the first model is a programming language model, the neural network compiler's front-end, middle-end, and back-end components will compile the optimized model into a sequence of target instructions. Alternatively, if the first model is an intermediate representation model, the neural network compiler's back-end components and toolchain will be used to compile the optimized model into a sequence of target instructions.

[0064] The neural network model compilation method provided in this embodiment converts preset operations such as dot product, matrix multiplication, and linear operations, including multiply-accumulate operations, into convolution operations supported by the neural network processor during the model compilation process. This allows the generated target instruction sequence to perform preset operations through convolution operations, thereby accelerating the above-mentioned preset operations of the model through a neural network processor used to accelerate convolutional neural networks, and greatly improving the model inference speed.

[0065] Figure 3 This is a flowchart illustrating a method for compiling a neural network model provided in another exemplary embodiment of this disclosure.

[0066] In some optional embodiments, after obtaining the first model corresponding to the neural network model to be compiled in step 201, the method further includes:

[0067] Step 301: For any network layer of the first model, obtain the operation parameters of that network layer.

[0068] Operational parameters describe the parameters related to the network layer. These may include network layer identifier parameters, input parameters, computational parameters, output parameters, etc., and can be set according to actual needs. Operational parameters include computational information, which can be used to determine whether the network layer contains preset operations. For example, `conv` and `pointwiseconv` represent parameters for various convolutional operations (including kernel size, stride, etc.), and parameters for matrix multiplication (including first matrix, second matrix, matrix multiplication operation, etc.). Different types of network layers may include different operational parameters, which can be set according to actual needs.

[0069] In some optional embodiments, the operation parameters may be descriptive information related to the operator nodes of the computation subgraph of the network layer, used to identify the operation type corresponding to the operator node in the computation subgraph, and are not specifically limited.

[0070] Step 302: Determine the state in which the network layer contains preset operations based on the operation parameters and preset judgment rules.

[0071] The preset judgment rule can be set according to the description of the relevant information of the preset operation in the operation parameters. It is used to determine the state of any network layer containing the preset operation based on the operation parameters. This state can include a state of inclusion and a state of non-inclusion.

[0072] Step 303: In response to the state of including preset operations being the included state, determine that the network layer is a network layer including preset operations.

[0073] In some optional embodiments, each network layer of the first model can be traversed. For each traversed network layer, the state in which the network layer contains a preset operation is determined based on the operation parameters of the network layer. If the state is a state of inclusion, the network layer is determined to be a network layer containing the preset operation. Then, the optimization process of converting the preset operation into a convolution operation supported by the neural network processor is performed on the network layer to obtain the optimization result corresponding to the network layer. When the network layers of the first model have been traversed, the optimization of the first model is completed, and the optimized model corresponding to the first model is obtained.

[0074] This embodiment identifies network layers containing preset operations in the first model through preset judgment rules, providing an effective reference for determining network layers that need to be optimized.

[0075] Figure 4 This is a flowchart illustrating a method for compiling a neural network model provided in another exemplary embodiment of this disclosure.

[0076] In some optional embodiments, step 202 may specifically include the following steps:

[0077] Step 2021: Select any one of the network layers in the first model that contains a preset operation as the target network layer.

[0078] Step 2022: Determine the first input tensor and the second input tensor corresponding to the preset operation in the target network layer.

[0079] The determination of the first and second input tensors can be based on the input tensors of the target network layer and other operations preceding the preset operation in the target network layer. For example, if the preset operation is the first operation of the target network layer or the target network layer only includes this preset operation, then the first and second input tensors are determined based on the input tensors of the target network layer. If other operations precede the preset operation, and the input of the preset operation is the output of the other operations, then the first and second input tensors are determined based on the output tensors of the other operations. The method for determining the first and second input tensors can differ depending on the specific type of the preset operation. For example, for matrix multiplication, the preset operation inherently includes two tensors: a left matrix (first matrix) and a right matrix (second matrix), which can be used as the first and second input tensors corresponding to the preset operation, respectively. However, for linear operations, the input only includes one input tensor, and the linear transformation matrix used for the linear operation is the weight parameter obtained during the training of the target network layer; this linear transformation matrix needs to be used as the second input tensor. Therefore, the first input tensor and the second input tensor are input tensors corresponding to preset operations determined according to certain rules. They may not be the original input tensors of the preset operations, so that they can be used for transformation in subsequent operations.

[0080] In some optional embodiments, the number of first input tensors can be one or more depending on the dimension of the original input tensor of the preset operation. Similarly, the number of second input tensors can be one or more.

[0081] For example, in batch matrix multiplication, if the original input tensor has a batch matrix of dimension B*M*K (i.e., B left matrices) and a batch matrix of dimension B*K*N (i.e., B right matrices), then the number of first input tensors is B, and the number of second input tensors is B. The first and second input tensors correspond one-to-one and are used for B matrix multiplication operations. Each matrix multiplication operation is converted into a convolution operation. Therefore, batch matrix multiplication is converted into B convolution operations. For preset operations on original input tensors with more dimensions, the first and second input tensors can be determined based on the specific characteristics of the operation and the characteristics of the convolution operation. For example, if the original input tensors are I*J*M*T and K*T*N, I*J*M*T represents an I-row, J-column M*T matrix array (including I*J left matrices of height M and width T), and K*T*N represents K T*N right matrices, each of height T and width N. If I*J left M*T matrices are multiplied by K right T*N matrices respectively, resulting in I*J*K M*N result matrices, then the number of first input tensors corresponding to this preset operation is determined to be I*J, and the number of second input tensors is determined to be K. Each first input tensor corresponds to each second input tensor for matrix multiplication, thus determining that I*J*K matrix multiplication operations are required. Each matrix multiplication operation is converted into a corresponding convolution operation, thus the preset operation needs to be converted into I*J*K convolution operations.

[0082] Step 2023: Based on the first transformation rule, perform data structure transformation on the first input tensor to obtain the transformed third input tensor.

[0083] The first transformation rule can be set according to the characteristics of the preset operation and the characteristics of the convolution operation. The setting principle is to make the third input tensor obtained by the transformation can be used as the input tensor of the convolution operation, and to obtain the operation result of the preset operation through the convolution operation. The specific transformation rule is not limited.

[0084] For example, taking matrix multiplication as an example, the first input tensor is a left matrix, which is a two-dimensional tensor. The input tensor for convolution operations supported by the neural network processor is a four-dimensional tensor. Therefore, the two-dimensional matrix is ​​converted into a four-dimensional tensor suitable for this convolution operation. For example, the first input tensor M*K (where M represents the height of the first input tensor and K represents the width of the first input tensor) is transformed into a third input tensor of 1*1*M*K. In 1*1*M*K, the first 1 represents the number of three-dimensional feature data included in the third input tensor, the second 1 represents the height of the three-dimensional feature data of the third input tensor, M represents the width of the three-dimensional feature data of the third input tensor, and K represents the number of channels of the three-dimensional feature data of the third input tensor.

[0085] Step 2024: Based on the second transformation rule, perform data structure transformation on the second input tensor to obtain the transformed fourth input tensor.

[0086] The second transformation rule can be set according to the characteristics of the preset operation and the convolution operation. The setting principle is to make the fourth input tensor obtained by the transformation can be used as the weight tensor of the convolution operation (or the convolution weight tensor), and to obtain the operation result of the preset operation through the convolution operation. The specific transformation rule is not limited.

[0087] For example, taking matrix multiplication as an example, the corresponding second input tensor is a right matrix, such as a K*N matrix. The convolution weight tensor of pointwise convolution operation includes one or more convolution kernels. Each convolution kernel has a dimension of 1*1*C three-dimensional data, where the first 1 represents the height of the convolution kernel, the second 1 represents the width of the convolution kernel, and C represents the number of channels of the convolution kernel. Then, the convolution weight tensor is a four-dimensional tensor of N*1*1*C, where N represents the number of convolution kernels, i.e., N 1*1*C three-dimensional convolution kernels. Based on this, and combining the specific operation of convolution operation with the relevant characteristics of matrix multiplication operation, the two-dimensional second input tensor is transformed into a fourth input tensor, so that the fourth input tensor can be used as a weight tensor to convolve with the above-mentioned third input tensor to obtain the output tensor corresponding to the preset operation. For example, the number of channels C of the convolution weight tensor is the number of rows K of the second input tensor, and the number of convolution kernels of the convolution weight tensor is the number of columns N of the second input tensor.

[0088] Steps 2023 and 2024 are not in any particular order.

[0089] Step 2025: Use the third input tensor as the input tensor for the convolution operation, and use the fourth input tensor as the convolution weight tensor for the convolution operation.

[0090] The convolution weight tensor includes one or more convolution kernels for performing convolution operations on the third input tensor. The multiplication and accumulation operations of the preset operations are realized through the convolution operation of each convolution kernel, and the preset operations such as matrix multiplication, dot product, and linear operation are realized through one or more convolution kernels.

[0091] Step 2026: Perform convolution operation based on the third input tensor and the convolution weight tensor to obtain the first output tensor.

[0092] The convolution operation is a pointwise convolution operation. The third input tensor is weighted along the depth direction (channel direction) by one or more convolution kernels included in the convolution weight tensor, resulting in a first output tensor with the same number of channels as the number of convolution kernels. The dimensions of the first output tensor are H*W*N, where H represents height, W represents width, and N represents the number of channels, which also represents the number of convolution kernels in the convolution weight tensor. Since pointwise convolution does not change the size of the input tensor in the height and width dimensions, only changing the number of channels, H*W is the same size as the third input tensor.

[0093] In some optional embodiments, there can be multiple first input tensors. For example, if multiple matrix multiplication operations between left matrices and the same right matrix are required, then there are multiple first input tensors. Each first input tensor can be transformed according to the above transformation rules. Alternatively, multiple first input tensors can be transformed together as a batch. In this case, the multiple first input tensors can be represented as an S*M*K data structure, transformed into an S*1*M*K third input tensor, and convolved with an N*1*1*K convolution weight tensor to obtain S 1*M*N first output tensors.

[0094] In some optional embodiments, the number of both the first and second input tensors is multiple and the same. For example, in batch matrix multiplication, it is necessary to perform matrix multiplications of a batch of B*M*K matrices (i.e., B left matrices) with a corresponding batch of B*K*N matrices (i.e., B right matrices) to obtain a batch of B*M*N matrices. Therefore, the batch matrix multiplication can be split into B matrix multiplications and processed according to the matrix multiplication operation flow described above.

[0095] In some optional embodiments, the number of first input tensors and the number of second input tensors are both multiple and different. For example, in the example above, the original input tensors are I*J M*T left matrices determined by I*J M*T tensors and K*T*N tensors, which are used as I*J first input tensors and K T*N right matrices as K second input tensors, forming I*J*K input tensor pairs. Each input tensor pair includes a first input tensor and a second input tensor, corresponding to a matrix multiplication operation. The first and second input tensors of each input tensor pair are transformed according to the above steps to obtain the corresponding third and fourth input tensors, thereby obtaining I*J*K first output tensors.

[0096] In practical applications, when converting a single predefined operation into multiple convolution operations, these operations can be executed in parallel or sequentially, depending on the scheduling rules of the neural network processor. For example, if the neural network processor has sufficient resources, it can allocate computational resources for multiple convolution operations and execute them in parallel.

[0097] Step 2027: Based on the first output tensor, determine the second output tensor for the preset operation.

[0098] The second output tensor can be obtained by transforming the data structure of the first output tensor. Since the first output tensor obtained from convolution is typically a three-dimensional or four-dimensional tensor, while the output tensor of the preset operation is a one-dimensional or two-dimensional tensor, or the actual data items in each dimension of the data structure of the output tensor of the preset operation differ from those of the first output tensor of the convolution operation—for example, the number of channels in the convolution result might correspond to the width of the output result of the preset operation, and the width of the convolution result might correspond to the height of the output result of the preset operation—the first output tensor is transformed into a tensor with the same data structure as the output tensor of the preset operation through data structure transformation. The specific transformation from the first output tensor to the second output tensor can be determined based on the specific preset operation, and this disclosure does not impose any limitations.

[0099] For example, for matrix multiplication of two matrices M*K (i.e., height * width) and K*N (i.e., height * width), the result of matrix multiplication is an M*N (i.e., height * width) matrix. The first output tensor obtained by convolution is a 1*M*N (i.e., height * width * number of channels) tensor. The result matrix of the M*N (i.e., height * width) matrix multiplication operation is obtained by data structure transformation.

[0100] Step 2028: Based on the second output tensor, determine the target output tensor of the first network layer to obtain the first network layer.

[0101] If the preset operation is the last or only operation of the target network layer, then the second output tensor can be used as the output tensor of the target network layer, which is also the target output tensor of the optimized first network layer. After the above transformation, the target network layer is transformed into the first network layer based on convolution operation.

[0102] This embodiment determines the first and second input tensors for a preset operation, and then performs data structure transformations on the first and second input tensors according to different transformation rules. This allows the obtained third and fourth input tensors to be used as the input tensor and convolution weight tensor for convolution operations, respectively. The result of the preset operation is obtained through convolution operations, thus laying the foundation for accelerating the preset operation using a neural network processor that accelerates convolution operations, and helping to further improve the model inference speed.

[0103] In some optional embodiments, step 2022, determining the first and second input tensors corresponding to the preset operations in the target network layer, includes:

[0104] Determine the operation type of the preset operation; in response to the preset operation type being dot product operation, use the first tensor and the second tensor to be subjected to dot product operation as the first input tensor and the second input tensor, respectively; or, in response to the preset operation type being matrix multiplication operation, use the first matrix and the second matrix to be subjected to matrix multiplication operation as the first input tensor and the second input tensor, respectively; or, in response to the preset operation type being linear operation, use the original input tensor of the linear operation as the first input tensor and the linear transformation matrix of the linear operation as the second input tensor.

[0105] In this model, the first and second tensors are one-dimensional tensors, i.e., vectors. The first and second matrices are two-dimensional tensors. Matrix multiplication operations can include two-dimensional matrix multiplication and multidimensional matrix multiplication. Depending on the dimension of the matrix multiplication operation, the number of the first and second matrices can be different. Examples include multidimensional matrix multiplication operations such as batch matrix multiplication, multidimensional matrix multiplication operations of I*J*M*T tensors and K*T*N tensors, and so on. Dot product operations can also be called one-dimensional matrix multiplication operations. For specific dot product operations, matrix multiplication operations, and linear operations, please refer to the previous content, which will not be repeated here. The linear transformation matrix is ​​the weight parameter obtained through model training. Linear operations, for example, are in fully connected layers of the model, where each network node in the current layer is obtained by performing a linear operation (weighted sum) on the outputs of the nodes in the previous layer according to their corresponding weights. Dot product operations, for example, are in self-attention network layers of the model, which require calculating the dot product of the query vector and the key vector. Matrix multiplication operations, such as those in self-attention network layers in a model, require multiplying the input tensor with mapping tensors (which can include query mapping tensors, key mapping tensors, and value mapping tensors) to obtain query tensors, key tensors, and value tensors.

[0106] This embodiment determines the first and second input tensors corresponding to preset operations by setting preset operations for different operation types. This can convert matrix multiplication, dot product, and linear operations into pointwise convolution operations supported by the neural network processor. As a result, the matrix multiplication, dot product, and linear operations in the model can be accelerated based on the neural network processor, effectively improving the model inference speed.

[0107] In some optional embodiments, step 2023, based on the first transformation rule, performs a data structure transformation on the first input tensor to obtain the transformed third input tensor, including:

[0108] The data structure of the first input tensor is transformed from M*K to 1*1*M*K to obtain the third input tensor, where M represents the number of rows of the first input tensor and K represents the number of columns of the first input tensor.

[0109] When M is greater than 1, the first input tensor can represent the first matrix of the matrix multiplication operation. When M equals 1 and K is greater than 1, it can represent the first tensor of the dot product operation or the input tensor of the linear operation.

[0110] In an optional example, Figure 5 This is a schematic diagram illustrating the data structure transformation principle of the first input tensor provided in an exemplary embodiment of this disclosure. In this example, M = 4, K = 6, that is, the first input tensor is a two-dimensional tensor with 4 rows and 6 columns. The aij (i = 1, 2, ..., M, j = 1, 2, ..., K) in the gray box represents the element in the i-th row and j-th column of the first input tensor. The transformed third input tensor is a four-dimensional tensor of 1*H1*W1*C1. In 1*H1*W1*C1, 1 represents Batch, that is, the number of first input tensors M*K. H1 represents the height of the third input tensor, H1 = 1. W1 represents the width of the third input tensor, W1 = M = 4. C1 represents the number of channels of the third input tensor, C1 = K = 6. As can be seen, the six channels at each position in the third input tensor include the six elements of the corresponding row of the first input tensor. For example, position 11 includes the six elements a11, a12, ..., a16 in the first input tensor along the channel direction. During convolution, these elements are multiplied and accumulated with the convolution kernel of each six-channel convolution weight tensor.

[0111] This embodiment transforms the data structure of the first input tensor from M*K to 1*1*M*K, so that the number of channels in the transformed third input tensor is K. During the convolution operation, the K elements of each row of the first input tensor are multiplied and accumulated with the K elements of the convolution kernel formed by each column of the second input tensor. The result of the multiplication and accumulation of each row of the first input tensor and each column of the second input tensor is obtained. This result is an element of the matrix resulting from the multiplication of the first input tensor and the second input tensor matrix, thus preparing for the preset operation based on the convolution operation.

[0112] In some optional embodiments, step 2024, based on the second transformation rule, performs a data structure transformation on the second input tensor to obtain the transformed fourth input tensor, including:

[0113] The data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain the fourth input tensor. In K*N, K represents the number of rows (i.e., height) of the second input tensor, and N represents the number of columns (i.e., width) of the second input tensor. In N*1*1*K, N represents the number of three-dimensional data included in the fourth input tensor, 1*1 represents the height * width of the fourth input tensor, and K represents the number of channels of the fourth input tensor.

[0114] When N is greater than 1, the second input tensor can represent the second matrix of the matrix multiplication operation. When N = 1 and K is greater than 1, it can represent the first tensor of the dot product operation or the input tensor of the linear operation.

[0115] In an optional example, Figure 6 This is a schematic diagram illustrating the data structure transformation principle of the second input tensor provided in an exemplary embodiment of this disclosure. (Similar to the above...) Figure 5 Corresponding to the first input tensor, in this example, the second input tensor is a two-dimensional tensor with K rows and N columns of 3. bjn (j = 1, 2, ..., K, n = 1, 2, ..., N) represents the element in the j-th row and n-th column of the second input tensor. The fourth input tensor obtained by the data structure transformation is a four-dimensional tensor of N*H2*W2*C2, where N represents the batch size of the fourth input tensor, H2 represents the height of the fourth input tensor, W2 represents the width of the fourth input tensor, and C2 represents the number of channels of the fourth input tensor. It can be seen that the second input tensor is transformed into a fourth input tensor comprising three 1*1*6 vectors. Each 1*1*6 vector can be used as a convolution kernel to perform a convolution operation with the aforementioned third input tensor.

[0116] This embodiment transforms the second input tensor from K*N to N*1*1*K, so that the obtained fourth input tensor can be used as N 1*1*K convolution kernels to perform convolution operations with the aforementioned third input tensor. The convolution kernel includes K elements of one column of the second input tensor in the channel direction. During the convolution operation, it is multiplied and accumulated with the K channel elements of each position of the third input tensor, realizing the multiplication and accumulation of each row of the first input tensor and each column of the second input tensor. Based on this, the operation result of the preset operation is effectively obtained through the convolution operation, which facilitates the acceleration of the preset operation by the neural network processor and improves the model inference speed.

[0117] In some optional embodiments, the data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain the fourth input tensor, including:

[0118] The data structure of the second input tensor is transformed from K*N to K*1*1*N to obtain the first intermediate tensor; the first intermediate tensor is used as the weight to perform a convolution operation on the preset unit tensor to obtain the fourth input tensor of N*1*1*K.

[0119] The preset unit tensor is a 1*1*N*N unit tensor obtained by transforming the data structure based on the N*N identity matrix.

[0120] In an optional example, Figure 7This is a schematic diagram illustrating the data structure transformation principle of the second input tensor provided in another exemplary embodiment of this disclosure. In this example, the K*N (i.e., height * width) two-dimensional second input tensor is transformed into a K*1*1*N (i.e., number * height * width * number of channels) first intermediate tensor, which serves as K 1*1*N convolution kernels. These kernels are then used for pointwise convolution with a preset unit tensor of 1*1*N*N (i.e., number * height * width * number of channels) corresponding to an N*N (i.e., height * width * number of channels) identity matrix to obtain an N*1*1*K (i.e., number * height * width * number of channels) fourth input tensor.

[0121] It should be noted that transforming the tensor data structure only changes the arrangement rules of the elements in the tensor, without changing the specific element values. This allows the transformed data structure to adapt to the data structure requirements of the corresponding operations of the neural network model, so that the neural network processor can accurately load the corresponding data to complete the model's inference.

[0122] In an optional example, Figure 8 This is a schematic diagram illustrating the convolution operation principle of a third input tensor and a convolution weight tensor provided in an exemplary embodiment of this disclosure. The convolution weight tensor includes N 1*1*K (i.e., height * width * number of channels) convolution kernels, which are convolved point-by-point with the third input tensor to obtain a 1*M*N first output tensor. Here, 1 represents the height of the first output tensor, M represents the width of the first output tensor, and N represents the number of channels in the first output tensor. Based on the first output tensor, a second output tensor of M*N (height * width) is obtained, which is the result of the preset operation. `cin` (i = 1, 2, ..., M, n = 1, 2, ..., N) represents the element in the i-th row and n-th column of the second output tensor. Taking `c11` as an example (other `cin` values ​​are similar), `c11` = a11*b11 + a12*b21 + ... + a1K*bK1. It can be seen that the second output tensor is the result of the matrix multiplication operation between the first and second input tensors. When M=N=1, the second output tensor can be the result of vector dot product or linear operation, which will not be elaborated further.

[0123] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0124] The compilation method for any neural network model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the compilation method for any neural network model provided in this disclosure can be executed by a processor, such as by a processor executing the compilation method for any neural network model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0125] Figure 9 This is a flowchart illustrating the inference method of a neural network model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 9 As shown, it includes the following steps:

[0126] Step 401: Obtain the input tensor corresponding to the neural network model and the target instruction sequence corresponding to the neural network model.

[0127] The target instruction sequence is obtained based on the compilation method of the neural network model provided in any of the above embodiments.

[0128] Step 402: The input tensor and the target instruction sequence are transmitted to the neural network processor.

[0129] The neural network processor may include a computing array for performing various neural network operations. The computing array may include multi-row and multi-column computing units, such as multipliers, adders, multiply-accumulators, etc., which can be configured according to actual needs.

[0130] In some optional embodiments, the transmission of the input tensor and the target instruction sequence can be achieved by writing the input tensor and the target instruction sequence into a storage unit within the neural network processor or to a storage device external to the neural network processor, and transmitting the storage address to the neural network processor, which then reads the input tensor and the target instruction sequence from the corresponding storage area. The specific method of invoking the neural network processor is not limited.

[0131] In some optional embodiments, the number of neural network processors can be one or more. For a target instruction sequence corresponding to a neural network model, multiple neural network processors can be invoked to implement the model's inference based on different network layers or different computational functions, thereby improving processing efficiency or increasing the utilization rate of the neural network processors. For example, when multiple or a large number of images need to be detected, certain scheduling rules can be set based on load balancing or utilization improvement rules to schedule multiple neural network accelerators to work collaboratively. The specific rules can be set according to actual needs.

[0132] Step 403: Process the input tensor based on the target instruction sequence executed by the neural network processor to obtain the inference result corresponding to the input tensor.

[0133] The neural network processor executes the target instruction sequence and, according to the optimized instruction sequence, performs the original preset operations of the neural network model based on convolutional operations during the inference process. It obtains the corresponding operation results and, combined with other inference processes, obtains the final inference result, further improving the model's inference speed. In relevant application scenarios, this effectively enhances the real-time performance. For example, in autonomous driving scenarios, it can significantly improve the speed of image processing such as object detection and semantic segmentation. For environmental perception in autonomous driving, it can provide more real-time perception results, enabling the vehicle to detect anomalies more promptly and take appropriate control measures earlier, thus improving vehicle safety. Alternatively, in intelligent cockpit scenarios, it can improve the real-time performance of the user's perception results, allowing for more timely responses to user interactions and enhancing the user experience.

[0134] The inference method of any neural network model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, the inference method of any neural network model provided in this disclosure can be executed by a processor, such as by a processor executing the inference method of any neural network model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0135] Exemplary device

[0136] Figure 10 This is a schematic diagram of the structure of a compilation apparatus for a neural network model provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement corresponding neural network model compilation method embodiments of this disclosure, such as... Figure 10 The device shown includes: a first acquisition module 501, a first processing module 502, a second processing module 503, and a third processing module 504.

[0137] The first acquisition module 501 is used to acquire the first model corresponding to the neural network model to be compiled.

[0138] The first processing module 502 is used to take any one of the network layers in the first model that contains a preset operation as the target network layer, convert the preset operation of the target network layer into a convolution operation supported by the neural network processor, and obtain the converted first network layer. The preset operation is an operation that includes multiplication and accumulation operations.

[0139] The second processing module 503 is used to obtain the optimized model corresponding to the first model based on the transformed first network layer.

[0140] The third processing module 504 is used to generate, based on the optimization model, the target instruction sequence that can be executed by the neural network processor corresponding to the neural network model to be compiled.

[0141] Figure 11 This is a schematic diagram of the structure of a compilation device for a neural network model provided in another exemplary embodiment of this disclosure.

[0142] In some optional embodiments, the apparatus of this disclosure further includes: a second acquisition module 601, a first determination module 602, and a second determination module 603.

[0143] The second acquisition module 601 is used to acquire the operation parameters of any network layer of the first model.

[0144] The first determining module 602 is used to determine the state in which the network layer contains preset operations based on the operation parameters and preset judgment rules.

[0145] The second determining module 603 is used to determine the network layer as a network layer containing the preset operation in response to the state containing the preset operation being a state containing the preset operation.

[0146] Figure 12 This is a schematic diagram of the structure of the first processing module 502 provided in an exemplary embodiment of this disclosure.

[0147] In some optional embodiments, the first processing module 502 includes:

[0148] The first determining unit 5021 is used to select any one of the network layers in the first model that contains a preset operation as the target network layer.

[0149] The second determining unit 5022 is used to determine the first input tensor and the second input tensor corresponding to the preset operation in the target network layer.

[0150] The first processing unit 5023 is used to perform data structure transformation on the first input tensor based on the first transformation rule to obtain the transformed third input tensor.

[0151] The second processing unit 5024 is used to perform data structure transformation on the second input tensor based on the second transformation rule to obtain the transformed fourth input tensor.

[0152] The third determining unit 5025 is used to use the third input tensor as the input tensor of the convolution operation and the fourth input tensor as the convolution weight tensor of the convolution operation.

[0153] The third processing unit 5026 is used to perform convolution operations based on the third input tensor and the convolution weight tensor to obtain the first output tensor.

[0154] The fourth processing unit 5027 is used to determine the second output tensor of the preset operation based on the first output tensor.

[0155] The fifth processing unit 5028 is used to determine the target output tensor of the first network layer based on the second output tensor, and obtain the first network layer.

[0156] In some optional embodiments, the second determining unit 5022 is specifically used for:

[0157] Determine the operation type of the preset operation; in response to the preset operation type being dot product operation, use the first tensor and the second tensor to be subjected to dot product operation as the first input tensor and the second input tensor, respectively; or, in response to the preset operation type being matrix multiplication operation, use the first matrix and the second matrix to be subjected to matrix multiplication operation as the first input tensor and the second input tensor, respectively; or, in response to the preset operation type being linear operation, use the original input tensor of the linear operation as the first input tensor and the linear transformation matrix of the linear operation as the second input tensor.

[0158] In some optional embodiments, the first processing unit 5023 is specifically used for:

[0159] The data structure of the first input tensor is transformed from M*K to 1*1*M*K to obtain the third input tensor, where M represents the number of rows of the first input tensor and K represents the number of columns of the first input tensor.

[0160] In some optional embodiments, the second processing unit 5024 is specifically used for:

[0161] The data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain the fourth input tensor, where K represents the number of rows of the second input tensor and N represents the number of columns of the second input tensor.

[0162] In some optional embodiments, the second processing unit 5024 is specifically used for:

[0163] The data structure of the second input tensor is transformed from K*N to K*1*1*N to obtain the first intermediate tensor; the first intermediate tensor is used as the weight to perform a convolution operation on the preset unit tensor to obtain the fourth input tensor of N*1*1*K.

[0164] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0165] Figure 13This is a schematic diagram of the structure of an inference apparatus for a neural network model provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement the corresponding inference method embodiment of the neural network model of this disclosure, such as... Figure 13 The apparatus shown includes:

[0166] The second acquisition module 601 is used to acquire the input tensor corresponding to the neural network model and the target instruction sequence corresponding to the neural network model.

[0167] The target instruction sequence is obtained based on the compilation method of the neural network model provided in any of the above embodiments.

[0168] The transmission module 602 is used to transmit the input tensor and the target instruction sequence to the neural network processor 603.

[0169] The neural network processor 603 is used to execute a target instruction sequence to process the input tensor and obtain the inference result corresponding to the input tensor.

[0170] The number of neural network processors 603 can be one or more, which can be set according to actual needs.

[0171] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0172] Exemplary electronic devices

[0173] Figure 14 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure, including at least one processor 11 and a memory 12.

[0174] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0175] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of this disclosure described above.

[0176] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0177] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0178] The output device 14 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0179] Of course, for the sake of simplicity, Figure 14 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0180] Exemplary computer program products and computer-readable storage media

[0181] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods in the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0182] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods in the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0184] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0185] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0186] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for compiling a neural network model, comprising: Obtain the first model corresponding to the neural network model to be compiled; Take any one of the network layers in the first model that contains a preset operation as the target network layer, and convert the preset operation of the target network layer into a convolution operation supported by the neural network processor to obtain the converted first network layer. The preset operation is an operation that includes multiply-accumulate operations. Based on the transformed first network layer, an optimized model corresponding to the first model is obtained; Based on the optimization model, a target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled is generated.

2. The method according to claim 1, wherein, The step of converting the preset operation of the target network layer into a convolution operation supported by the neural network processor to obtain the converted first network layer includes: Determine the first input tensor and the second input tensor corresponding to the preset operation; Based on the first transformation rule, the first input tensor is subjected to a data structure transformation to obtain the transformed third input tensor; Based on the second transformation rule, the second input tensor is subjected to a data structure transformation to obtain the transformed fourth input tensor; Use the third input tensor as the input tensor for the convolution operation; The fourth input tensor is used as the convolution weight tensor for the convolution operation; A convolution operation is performed based on the third input tensor and the convolution weight tensor to obtain the first output tensor; Based on the first output tensor, determine the second output tensor of the preset operation; Based on the second output tensor, the target output tensor of the first network layer is determined, and the first network layer is obtained.

3. The method according to claim 2, wherein, The step of performing a data structure transformation on the first input tensor based on the first transformation rule to obtain the transformed third input tensor includes: The data structure of the first input tensor is transformed from M*K to 1*1*M*K to obtain the third input tensor, where M represents the number of rows of the first input tensor and K represents the number of columns of the first input tensor.

4. The method according to claim 2, wherein, The step of performing a data structure transformation on the second input tensor based on the second transformation rule to obtain the transformed fourth input tensor includes: The data structure of the second input tensor is transformed from K*N to N*1*1*K to obtain the fourth input tensor, where K represents the number of rows of the second input tensor and N represents the number of columns of the second input tensor.

5. The method according to claim 4, wherein, The step of transforming the data structure of the second input tensor from K*N to N*1*1*K to obtain the fourth input tensor includes: Transform the data structure of the second input tensor from K*N to K*1*1*N to obtain the first intermediate tensor; Using the first intermediate tensor as weights, a convolution operation is performed on the preset unit tensor to obtain the fourth input tensor of N*1*1*K.

6. The method according to claim 2, wherein, Determining the first input tensor and the second input tensor corresponding to the preset operation includes: Determine the operation type of the preset operation; In response to the preset operation type being dot product, the first tensor and the second tensor to be subjected to the dot product operation are respectively used as the first input tensor and the second input tensor; or, In response to the preset operation type being matrix multiplication, the first matrix and the second matrix to be multiplied are respectively used as the first input tensor and the second input tensor; or, In response to the preset operation being a linear operation, the original input tensor of the linear operation is used as the first input tensor, and the linear transformation matrix of the linear operation is used as the second input tensor.

7. The method according to claim 1, after obtaining the first model corresponding to the neural network model to be compiled, further comprising: For any network layer of the first model, obtain the operation parameters of that network layer; Based on the operation parameters and preset judgment rules, the state in which the network layer contains the preset operation is determined; In response to the state being an inclusive state, the network layer is determined to be a network layer that includes the preset operation.

8. A reasoning method for a neural network model, comprising: Obtain the input tensor corresponding to the neural network model and the target instruction sequence corresponding to the neural network model; The target instruction sequence is obtained based on the compilation method of the neural network model according to any one of claims 1-7; The input tensor and the target instruction sequence are transmitted to the neural network processor; The neural network processor executes the target instruction sequence to process the input tensor, thereby obtaining the inference result corresponding to the input tensor.

9. A compilation device for a neural network model, comprising: The first acquisition module is used to acquire the first model corresponding to the neural network model to be compiled. The first processing module is configured to take any one of the network layers in the first model that contains a preset operation as the target network layer, convert the preset operation of the target network layer into a convolution operation supported by the neural network processor, and obtain the converted first network layer, wherein the preset operation is an operation including multiplication and accumulation operations. The second processing module is used to obtain the optimized model corresponding to the first model based on the transformed first network layer; The third processing module is used to generate, based on the optimized model, the target instruction sequence executable by the neural network processor corresponding to the neural network model to be compiled.

10. An inference device for a neural network model, comprising: The second acquisition module is used to acquire the input tensor corresponding to the neural network model and the target instruction sequence corresponding to the neural network model. The target instruction sequence is obtained based on the compilation method of the neural network model according to any one of claims 1-7; The transmission module is used to transmit the input tensor and the target instruction sequence to the neural network processor; The neural network processor is used to execute the target instruction sequence to process the input tensor and obtain the inference result corresponding to the input tensor.

11. A computer-readable storage medium storing a computer program for executing a method for compiling a neural network model according to any one of claims 1-7.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the compilation method of the neural network model according to any one of claims 1-7.

Images