Convolution operation method and device, electronic equipment and readable storage medium

Abstract

This invention provides a convolution operation method, apparatus, electronic device, and readable storage medium. The method includes: acquiring input data and an operator stream file; the operator stream file includes operators; determining the operator type of the operators in the operator stream file; when the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator; and performing a convolution operation on the input data using a multiplication-accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator, to obtain a convolution operation result. The neural network processor of this invention includes accelerated hardware for convolution operations of the first specified convolution kernel type. Therefore, by using a multiplication-accumulation unit to perform convolution operations on the input data based on the first specified convolution kernel type and the convolution kernel type of the operator, the efficiency of convolution operations is improved, and the waste of hardware resources in the neural network processor is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a convolution operation method, a convolution operation device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rise of deep learning neural networks and the advent of the era of artificial intelligence and big data, CPUs (Central Processing Units) and GPUs (Graphics Processing Units) are gradually becoming insufficient to meet the needs of deep learning. Faced with the growing demand and the vast expected market, it is essential to design a high-efficiency intelligent processor specifically for neural network deep learning. Therefore, NPUs (Neural-network Processing Units) have emerged.

[0003] However, current neural network processor designs are not perfect. When using a neural network processor for convolution operations, some MAC (Multiply-Accumulate) units in the neural network processor are idle, resulting in low efficiency of convolution operations and wasting hardware resources. Summary of the Invention

[0004] The present invention provides a convolution operation method, apparatus, electronic device, and computer-readable storage medium to solve the problem of low efficiency in convolution operations of neural network processors and waste of hardware resources of neural network processors.

[0005] This invention discloses a convolution operation method, relating to a neural network processor. The neural network processor includes a multiplication-accumulation unit, which further includes an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type. The method includes:

[0006] Obtain input data and an operator stream file; the operator stream file includes operators;

[0007] Determine the operator type of the operator in the operator stream file;

[0008] When the operator type of the operator is a specified operator type, the convolution kernel type of the operator is determined;

[0009] Based on the first specified convolution kernel type and the convolution kernel type of the operator, the input data is convolved using the corresponding multiplication and accumulation unit to obtain the convolution operation result.

[0010] Optionally, determining the operator type of the operator in the operator stream file includes:

[0011] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, the operator type of the operator is determined to be a hardware operator;

[0012] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0013] Optionally, when the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator includes:

[0014] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and the convolution kernel type of the operator is determined.

[0015] Optionally, based on the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result, including:

[0016] When the kernel type of the operator is not the first specified kernel type, determine whether the kernel type of the operator is the second specified kernel type;

[0017] When the convolution kernel type of the operator is the second specified convolution kernel type, the input channels of the input data are merged for each preset number of input channels to obtain merged input data;

[0018] Determine the number of merged input channels for the merged input data;

[0019] The merged input data is convolved according to the preset stride, the number of merged input channels, and the first specified convolution kernel type to obtain the convolution result.

[0020] Optionally, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset stride includes 3.

[0021] Optionally, based on the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result, including:

[0022] When the convolution kernel type of the operator is the first specified convolution kernel type, the number of input channels of the input data is determined;

[0023] According to the preset stride, the number of input channels, and the first specified convolution kernel type, the multiplication and accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result.

[0024] Optionally, after determining the operator type of the operator in the operator stream file, the method further includes:

[0025] When the operator type of the operator is not the specified operator type, the preset software algorithm of the operator is invoked to perform convolution operation on the input data.

[0026] Optionally, before acquiring the input data and operator stream file, the method further includes:

[0027] An operator stream file is generated using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a set of hardware instructions, and a set of quantized network weights, and the operator stream includes operators;

[0028] Store the operator stream file in a designated memory;

[0029] When the neural network processor starts, the operator stream file is loaded into random access memory to retrieve the operator stream file from the random access memory for convolution operations.

[0030] Optionally, after performing convolution operations on the input data using the multiplication-accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator to obtain the convolution operation result, the method further includes:

[0031] After the operator in the operator stream file has been executed once, the convolution operation result is converted into a specified format; the specified format includes at least a floating-point number format.

[0032] This invention also discloses a convolution operation apparatus, relating to a neural network processor, the neural network processor including a multiplication-accumulation unit, the neural network processor including an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type, the apparatus comprising:

[0033] The data acquisition module is used to acquire input data and operator stream files; the operator stream files include operators.

[0034] An operator type determination module is used to determine the operator type of the operator in the operator stream file;

[0035] A kernel type determination module is used to determine the kernel type of the operator when the operator type of the operator is a specified operator type;

[0036] The convolution operation module is used to perform convolution operation on the input data according to the first specified convolution kernel type and the convolution kernel type of the operator, and obtain the convolution operation result.

[0037] Optionally, the operator type determination module is used for:

[0038] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, the operator type of the operator is determined to be a hardware operator;

[0039] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0040] Optionally, the kernel type determination module is used for:

[0041] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and the convolution kernel type of the operator is determined.

[0042] Optionally, the convolution operation module is used for:

[0043] When the kernel type of the operator is not the first specified kernel type, determine whether the kernel type of the operator is the second specified kernel type;

[0044] When the convolution kernel type of the operator is the second specified convolution kernel type, the input channels of the input data are merged for each preset number of input channels to obtain merged input data;

[0045] Determine the number of merged input channels for the merged input data;

[0046] The merged input data is convolved according to the preset stride, the number of merged input channels, and the first specified convolution kernel type to obtain the convolution result.

[0047] Optionally, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset stride includes 3.

[0048] Optionally, the convolution operation module is used for:

[0049] When the convolution kernel type of the operator is the first specified convolution kernel type, the number of input channels of the input data is determined;

[0050] According to the preset stride, the number of input channels, and the first specified convolution kernel type, the multiplication and accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result.

[0051] Optionally, the apparatus further includes: a software operator convolution operation module, used for:

[0052] When the operator type of the operator is not the specified operator type, the preset software algorithm of the operator is invoked to perform convolution operation on the input data.

[0053] Optionally, the apparatus further includes: an operator stream file generation module, used for:

[0054] An operator stream file is generated using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a set of hardware instructions, and a set of quantized network weights, and the operator stream includes operators;

[0055] Store the operator stream file in a designated memory;

[0056] When the neural network processor starts, the operator stream file is loaded into random access memory to retrieve the operator stream file from the random access memory for convolution operations.

[0057] Optionally, the apparatus further includes: a format conversion module, used for:

[0058] After the operator in the operator stream file has been executed once, the convolution operation result is converted into a specified format; the specified format includes at least a floating-point number format.

[0059] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0060] The memory is used to store computer programs;

[0061] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0062] This invention also discloses a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in this invention.

[0063] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0064] The embodiments of the present invention have the following advantages:

[0065] In this embodiment of the invention, a neural network processor is involved. The neural network processor includes a multiplication-accumulation unit and accelerated hardware for convolution operations of a first specified convolution kernel type. During convolution operations, input data and an operator stream file are acquired, and the operator type of the operator in the operator stream file is determined. When the operator type is a specified operator type, the convolution kernel type of the operator is determined. Based on the first specified convolution kernel type and the operator's convolution kernel type, the multiplication-accumulation unit in the neural network processor is used to perform convolution operations on the input data to obtain the convolution operation result. Since the neural network processor in this embodiment of the invention includes accelerated hardware for convolution operations of the first specified convolution kernel type, using the multiplication-accumulation unit in the neural network processor to perform convolution operations on the input data based on the first specified convolution kernel type and the operator's convolution kernel type can improve the efficiency of convolution operations and avoid wasting hardware resources in the neural network processor. Attached Figure Description

[0066] Figure 1 This is a flowchart of the steps of a convolution operation method provided in an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the input channel merging of input data provided in this embodiment of the invention;

[0068] Figure 3 This is a schematic diagram of the convolution operation in a neural network processor provided in an embodiment of the present invention;

[0069] Figure 4 This is a structural block diagram of a convolution operation device provided in an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the hardware structure of an electronic device that implements various embodiments of the present invention. Detailed Implementation

[0071] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] In practical implementations, convolutional operations play a crucial role in tasks ranging from image classification and object detection to semantic segmentation, image generation, and style transfer. Currently, neural network processors (NPUs) typically employ hardware acceleration optimizations for certain convolutional operations, often focusing on 3x3 convolutions (using 3x3 kernels). Larger kernel operations are then achieved by stacking multiple 3x3 convolutions. For example, a 5x5 convolution can be implemented using four 3x3 convolutions, and a 7x7 convolution can be implemented using nine 3x3 convolutions. Since a 3x3 convolution hardware unit has nine MAC units, directly using these 3x3 units to perform a 1x1 convolution would leave eight MAC units idle, resulting in a waste of hardware resources (MAC units) in the neural network processor.

[0073] To address the aforementioned issues, this invention provides a convolution operation method that can perform convolution operations on input data and operator stream files using a specified convolution kernel type (e.g., 3x3 convolution operation) optimized for hardware acceleration in the neural network processor. This improves the efficiency of convolution operations and avoids energy waste caused by idle MAC units in the neural network processor.

[0074] Reference Figure 1 This diagram illustrates a flowchart of a convolution operation method provided in an embodiment of the present invention. The method relates to a neural network processor, which includes a multiplication-accumulation unit. The neural network processor includes an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type. Specifically, the method may include the following steps:

[0075] Step 101: Obtain input data and operator stream file; the operator stream file includes operators.

[0076] In its implementation, a Neural Processing Unit (NPU) is a hardware processor designed to accelerate neural network computations. Compared to a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU), the NPU is hardware-optimized for convolution operations of a first specified kernel type in a neural network. Therefore, when using the NPU to perform convolution operations of the first specified kernel type, the hardware resources within the NPU can be fully utilized, specifically the Multiply-Accumulate Unit (MAC unit). The MAC unit is a hardware unit used to perform multiplication and accumulation operations. In convolution operations, the MAC unit calculates the product between the convolution kernel and a local region of the input feature map (input data, which can be data related to the task to be performed, such as object detection, semantic segmentation, image generation, and style transfer), and then accumulates the product. The input data includes the number of input channels. One dimension of the number of input channels represents the number of feature maps in the input data. Each input channel corresponds to one feature map, which is a two-dimensional matrix obtained by sliding the convolution kernel across the input data and performing convolution operations. A convolutional kernel can be represented as a three-dimensional tensor, such as k*k*c, where k is the kernel size and c is the kernel depth, which must be the same as the number of input channels in the input data. The input data can be represented as a three-dimensional tensor, such as H*W*C, where H is the height, W is the width, and C is the number of input channels. The convolutional kernel slides across each input channel of the input data, performing a dot product operation at each position to extract local features from the input data. Then, the results from all channels are summed to obtain an element of the output feature map.

[0077] For example, the first specified convolution kernel type can be a 3x3 convolution, and the convolution operation of the first specified convolution kernel type is a convolution operation of a 3x3 convolution kernel. An operator stream file refers to a file format used to describe data processing flows in certain specific computing frameworks or systems. An operator stream file defines a series of operations (operators) and the data flow relationships between operators, thus forming a complete data processing or computation flow. Specifically, the neural network processor designs multiple multiplication-accumulation units to execute convolution operations of the first specified convolution kernel type, such as injecting 3x3 convolution operations, in parallel to improve the efficiency of convolution operations. Furthermore, a dedicated instruction set is designed for convolution operations of the first specified convolution kernel type to reduce instruction overhead, enabling the neural network processor to accelerate convolution operations and improve their efficiency when executing convolution operations of the first specified convolution kernel type.

[0078] In this embodiment of the invention, when it is necessary to perform convolution operations on the input data, the neural network processor can perform the corresponding convolution operation task according to the operator stream file.

[0079] Step 102: Determine the operator type of the operator in the operator stream file.

[0080] Step 103: When the operator type of the operator is a specified operator type, determine the convolution kernel type of the operator.

[0081] Step 104: Based on the first specified convolution kernel type and the convolution kernel type of the operator, perform convolution operation on the input data using the corresponding multiplication accumulation unit to obtain the convolution operation result.

[0082] In practical implementation, operators have corresponding convolution kernel types. For example, the convolution kernel type of an operator can be a 1x1 convolution kernel, a 3x3 convolution kernel, a 5x5 convolution kernel, etc. The multiplication-accumulation unit is used to perform the multiplication and accumulation operations in the convolution operation. For a convolution kernel, each element of the convolution kernel is multiplied by the corresponding element of the input data, and then all multiplications are accumulated. This requires multiple multiplication-accumulation units in the neural network processor to execute in parallel. For example, a 1x1 convolution kernel requires 1 multiplication-accumulation unit, a 3x3 convolution kernel requires 9 multiplication-accumulation units, and a 5x5 convolution kernel requires 25 multiplication-accumulation units.

[0083] In this embodiment of the invention, the operator type in the operator stream file can be of various types, such as software operators or hardware operators. A specified operator type refers to the type of operator that requires convolution operations using a neural network processor. Therefore, when the operator type in the operator stream file is a specified type, meaning that convolution operations need to be performed using multiplication-accumulation units in the neural network processor, it is necessary to further determine the convolution kernel type of the operator, and thus determine the number of multiplication-accumulation units in the neural network processor to be used. Specifically, in this embodiment of the invention, based on the first specified convolution kernel type and the convolution kernel type of the operator, the multiplication-accumulation units in the neural network processor can be used to perform convolution operations on the input data to obtain the convolution operation result.

[0084] The above convolution operation method involves a neural network processor, which includes a multiplication-accumulation unit and accelerated hardware for convolution operations of a first specified convolution kernel type. During convolution operations, input data and an operator stream file are acquired, and the operator type of the operator in the operator stream file is determined. When the operator type is a specified operator type, the convolution kernel type of the operator is determined. Based on the first specified convolution kernel type and the operator's convolution kernel type, the multiplication-accumulation unit in the neural network processor is used to perform convolution operations on the input data to obtain the convolution operation result. The neural network processor in this embodiment includes accelerated hardware for convolution operations of a first specified convolution kernel type. Therefore, by using the multiplication-accumulation unit in the neural network processor to perform convolution operations on the input data based on the first specified convolution kernel type and the operator's convolution kernel type, the efficiency of convolution operations can be improved, and the waste of hardware resources in the neural network processor can be avoided.

[0085] In one embodiment of the present invention, determining the operator type of the operator in the operator stream file includes:

[0086] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, the operator type of the operator is determined to be a hardware operator;

[0087] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0088] In practical implementation, a sequence of continuously executed hardware instructions is implemented as an OP_HW operator. That is, an operator that can be executed on a specified hardware platform is a hardware operator, which may include a neural network processor. All operations that cannot be implemented in hardware must be implemented as software operators.

[0089] In one embodiment of the present invention, after determining the operator type of the operator in the operator stream file, the method further includes:

[0090] When the operator type of the operator is not the specified operator type, the preset software algorithm of the operator is invoked to perform convolution operation on the input data.

[0091] In this embodiment of the invention, the operator stream file is parsed and the operators are executed sequentially. If it is determined that the operator type is a software operator, that is, the operator type is not the specified operator type, then the preset software algorithm corresponding to the operator is called to perform convolution operation on the input data, thereby obtaining the convolution operation result.

[0092] In one embodiment of the present invention, when the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator includes:

[0093] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and the convolution kernel type of the operator is determined.

[0094] In this embodiment of the invention, if it is determined that the operator type of the operator is a hardware operator, it can be determined that the operator type of the operator is a specified operator type. Then, it is necessary to further determine the convolution kernel type of the operator. Based on the convolution kernel type of the operator, combined with the first specified convolution kernel type that has been hardware optimized in the neural network processor, the corresponding multiplication and accumulation unit is used to perform convolution operation on the input data to obtain the convolution operation result, thereby improving the efficiency of the convolution operation on the input data.

[0095] In one embodiment of the present invention, according to the first specified convolution kernel type and the convolution kernel type of the operator, the input data is convolved using the multiplication-accumulation unit to obtain the convolution operation result, including:

[0096] When the kernel type of the operator is not the first specified kernel type, determine whether the kernel type of the operator is the second specified kernel type;

[0097] When the convolution kernel type of the operator is the second specified convolution kernel type, the input channels of the input data are merged for each preset number of input channels to obtain merged input data;

[0098] Determine the number of merged input channels for the merged input data;

[0099] The merged input data is convolved according to the preset stride, the number of merged input channels, and the first specified convolution kernel type to obtain the convolution result.

[0100] For example, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, the preset stride includes 3, and each preset number can be 9.

[0101] In this embodiment of the invention, when the convolution kernel type of the operator is not the first specified convolution kernel type, that is, when the convolution kernel type of the operator is not a 3*3 convolution kernel, the multiplication-accumulation unit of the first specified convolution kernel type in the neural network processor, which has been hardware optimized, cannot be used to accelerate the convolution operation. Therefore, at this time, it is further determined whether the convolution kernel type of the operator is the second specified convolution kernel type, that is, whether the convolution kernel type of the operator is a 1*1 convolution kernel. When the convolution kernel type of the operator is a 1*1 convolution kernel, every 9 input channels in the input data are merged to obtain merged input data. Then, the number of merged input channels of the merged input data is determined, and the multiplication-accumulation unit is used to accelerate the convolution operation on the merged input data according to the preset stride (e.g., 3), the number of merged input channels and the 3*3 convolution kernel to quickly obtain the convolution operation result.

[0102] In one embodiment of the present invention, according to the first specified convolution kernel type and the convolution kernel type of the operator, the input data is convolved using the multiplication-accumulation unit to obtain the convolution operation result, including:

[0103] When the convolution kernel type of the operator is the first specified convolution kernel type, the number of input channels of the input data is determined;

[0104] According to the preset stride, the number of input channels, and the first specified convolution kernel type, the multiplication and accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result.

[0105] When the operator's convolution kernel type is the first specified convolution kernel type, that is, when the operator's convolution kernel type is a 3*3 convolution kernel, the neural network processor is directly started, and the multiplication and accumulation unit is used to accelerate the convolution operation according to the preset stride, number of input channels and 3*3 convolution kernel, so as to quickly obtain the convolution operation result.

[0106] In one specific embodiment, after acquiring the input data and the operator stream file, the operator stream file is parsed, and the operators in the operator stream file are executed sequentially. If the operator is a software operator, the relevant software logic is called to perform the operation; if the operator is a hardware operator, the type of convolution kernel of the operator is determined. If the operator is a 3x3 convolution kernel, the hardware instruction parameters obtained from parsing the operator stream file are directly used to configure the neural network processor, and then the neural network processor is started to accelerate the convolution operation; if the operator is a 1x1 convolution kernel, every 9 input channels are merged into one channel, as detailed in [reference needed]. Figure 2 Then update the convolution operation channel data and row and column data according to the following formula:

[0107] (1) Calculate the number of channels after merging the 9 input channels: ch_num = (ch_num + 8) / 9;

[0108] (2) The number of rows after merging the 9 input channels is expanded to 3 times: hardware_in_row*=3;

[0109] (3) The number of columns after merging the 9 input channels is expanded to 3 times: hardware_in_col*=3;

[0110] (4) The stride after merging the 9 input channels is expanded from 1 to 3;

[0111] Then, the neural network processor is activated to accelerate the operation using a 3x3 convolution with a stride of 3.

[0112] For example, convolution is a multiplication-accumulation process, where each input channel's data undergoes matrix multiplication and then accumulation. If a 3x3 convolution operation's multiplication-accumulation unit is directly used to perform a 1x1 convolution operation, eight multiplication-accumulation units will be idle. To fully utilize hardware performance, every nine input channels are merged into one input channel. To avoid redundant computation, the convolution operation stride needs to be expanded from 1 to 3. For instance, a 1x1 convolution operation on 12 (number of rows) * 12 (number of columns) * 18 (number of input channels) of input data becomes a 3x3 convolution operation on 36 (number of rows) * 36 (number of columns) * 2 (number of input channels) of input data after merging.

[0113] In one embodiment of the present invention, before acquiring the input data and the operator stream file, the method further includes:

[0114] An operator stream file is generated using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a set of hardware instructions, and a set of quantized network weights, and the operator stream includes operators;

[0115] Store the operator stream file in a designated memory;

[0116] When the neural network processor starts, the operator stream file is loaded into random access memory to retrieve the operator stream file from the random access memory for convolution operations.

[0117] In this embodiment of the invention, an operator stream file is generated by encoding using an AI compiler (artificial intelligence compiler). The operator stream file generated by the AI ​​compiler is divided into four parts:

[0118] (1) The first part is the header, which has a fixed length. The definitions of each field in the header are shown in Table 1:

[0119] Table 1: All variable fields in the header

[0120]

[0121] (2) The second part is the operator stream, with the starting position as shown in the header field ops_offset. A sequence of hardware instructions executed continuously is implemented as the OP_HW operator (hardware operator). All operations that cannot be implemented by hardware must be implemented as software operators, and the running parameters are placed in the operator parameter list.

[0122] (3) The third part is the set of hardware instructions, and the starting position is shown in the header field cmd_offset in Table 1 above;

[0123] (4) The fourth part is the set of quantized network weights, with the starting position shown in the header field weight_offset in Table 1.

[0124] The operator stream file generated by the AI ​​compiler is permanently stored in a designated memory, such as flash memory. When the hardware system starts, the operator stream file is loaded into RAM (Random Access Memory) so that the application can access it.

[0125] In addition, in this embodiment of the invention, the decoding middleware is initialized to inform the AI ​​compiler of the location of the relevant operator stream file, the input data location of the AI ​​compiler, and the output data storage location.

[0126] In one embodiment of the present invention, after performing convolution operations on the input data using the multiplication-accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator, and obtaining the convolution operation result, the method may further include:

[0127] After the operator in the operator stream file has been executed once, the convolution operation result is converted into a specified format; the specified format includes at least a floating-point number format.

[0128] In this embodiment of the invention, after all operators in the operator stream file have been executed and the convolution result is obtained, the convolution result is converted to a specified format, and relevant post-processing operations for the AI ​​algorithm are performed. Specifically, the convolution result is usually in fixed-point format, and the convolution result can be converted from fixed-point format to floating-point format.

[0129] One of the inventive points of this invention is that it provides a method for accelerating 1x1 convolution operations. By rearranging the data, the nine input channels of the input data are merged into one channel, and then the input data after the channel merging is convolved with a 3x3 convolution kernel with a stride of 3. This improves the efficiency of the convolution operation of the 1x1 convolution kernel and avoids the waste of hardware resources caused by the idle multiplication and accumulation units in the neural network processor.

[0130] To enable those skilled in the art to better understand the embodiments of the present invention, a specific example is provided below. (Refer to...) Figure 3 This is a flowchart illustrating the convolution operation in a neural network processor according to an embodiment of the present invention. The specific steps include:

[0131] Step 301: Load the compiled operator stream file;

[0132] Step 302: Parse the operator stream file and initialize the context environment;

[0133] Step 303: Execute the operators in the operator stream file sequentially;

[0134] Step 304: Determine whether the operator is a hardware operator; if yes, proceed to step 305; otherwise, proceed to step 309.

[0135] Step 305: Determine if the convolution kernel is 1x1;

[0136] Step 306: Merge the input channels for the input data;

[0137] Step 307: Update the number of channels and the number of rows and columns for the convolution operation;

[0138] Step 308: Accelerate convolution operations in the 3x3 convolution multiplication and accumulation unit of the NPU;

[0139] Step 309: Execute the corresponding software algorithm;

[0140] Step 310: Store the calculation results of this layer (convolution operation results);

[0141] Step 311: Determine whether all operators in the operator stream file have been executed; if yes, proceed to step 312; otherwise, proceed to step 303.

[0142] Step 312: Export the final result.

[0143] By applying the embodiments of the present invention, the data of every 9 input channels is merged into one channel through data rearrangement, and then a 3x3 convolution operation with a stride of 3 is performed, which improves the efficiency of 1x1 convolution operation and avoids the energy waste caused by the idle multiplication and accumulation unit in the NPU.

[0144] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0145] Reference Figure 4 This diagram illustrates a structural block diagram of a convolution operation device provided in an embodiment of the present invention. The device relates to a neural network processor, which includes a multiplication-accumulation unit. The neural network processor further includes an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type. Specifically, the device may include the following modules:

[0146] Data acquisition module 401 is used to acquire input data and operator stream file; the operator stream file includes operators;

[0147] Operator type determination module 402 is used to determine the operator type of the operator in the operator stream file;

[0148] The kernel type determination module 403 is used to determine the kernel type of the operator when the operator type of the operator is a specified operator type;

[0149] The convolution operation module 404 is used to perform convolution operation on the input data according to the first specified convolution kernel type and the convolution kernel type of the operator, and obtain the convolution operation result.

[0150] In one embodiment of the present invention, the operator type determination module 402 is used for:

[0151] When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, the operator type of the operator is determined to be a hardware operator;

[0152] When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator.

[0153] In one embodiment of the present invention, the kernel type determination module 403 is used for:

[0154] When the operator type is a hardware operator, the operator type of the operator is determined to be a specified operator type, and the convolution kernel type of the operator is determined.

[0155] In one embodiment of the present invention, the convolution operation module 404 is used for:

[0156] When the kernel type of the operator is not the first specified kernel type, determine whether the kernel type of the operator is the second specified kernel type;

[0157] When the convolution kernel type of the operator is the second specified convolution kernel type, the input channels of the input data are merged for each preset number of input channels to obtain merged input data;

[0158] Determine the number of merged input channels for the merged input data;

[0159] The merged input data is convolved according to the preset stride, the number of merged input channels, and the first specified convolution kernel type to obtain the convolution result.

[0160] In one embodiment of the present invention, the first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset stride includes 3.

[0161] In one embodiment of the present invention, the convolution operation module 404 is used for:

[0162] When the convolution kernel type of the operator is the first specified convolution kernel type, the number of input channels of the input data is determined;

[0163] According to the preset stride, the number of input channels, and the first specified convolution kernel type, the multiplication and accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result.

[0164] In one embodiment of the present invention, the apparatus further includes: a software operator convolution operation module, used for:

[0165] When the operator type of the operator is not the specified operator type, the preset software algorithm of the operator is invoked to perform convolution operation on the input data.

[0166] In one embodiment of the present invention, the apparatus further includes: an operator stream file generation module, used for:

[0167] An operator stream file is generated using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a set of hardware instructions, and a set of quantized network weights, and the operator stream includes operators;

[0168] Store the operator stream file in a designated memory;

[0169] When the neural network processor starts, the operator stream file is loaded into random access memory to retrieve the operator stream file from the random access memory for convolution operations.

[0170] In one embodiment of the present invention, the apparatus further includes: a format conversion module, used for:

[0171] After the operator in the operator stream file has been executed once, the convolution operation result is converted into a specified format; the specified format includes at least a floating-point number format.

[0172] In this embodiment of the invention, a neural network processor is involved. The neural network processor includes a multiplication-accumulation unit and accelerated hardware for convolution operations of a first specified convolution kernel type. During convolution operations, input data and an operator stream file are acquired, and the operator type of the operator in the operator stream file is determined. When the operator type is a specified operator type, the convolution kernel type of the operator is determined. Based on the first specified convolution kernel type and the operator's convolution kernel type, the multiplication-accumulation unit in the neural network processor is used to perform convolution operations on the input data to obtain the convolution operation result. Since the neural network processor in this embodiment of the invention includes accelerated hardware for convolution operations of the first specified convolution kernel type, using the multiplication-accumulation unit in the neural network processor to perform convolution operations on the input data based on the first specified convolution kernel type and the operator's convolution kernel type can improve the efficiency of convolution operations and avoid wasting hardware resources in the neural network processor.

[0173] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0174] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described convolution operation method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0175] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described convolution operation method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0176] This invention also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described convolution operation method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0177] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0178] The electronic device 500 includes, but is not limited to, components such as: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will understand that... Figure 5 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In embodiments of the present invention, the electronic device includes, but is not limited to, mobile phones, tablet computers, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.

[0179] It should be understood that, in this embodiment of the invention, the radio frequency unit 501 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 510; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 501 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 501 can also communicate with networks and other devices through a wireless communication system.

[0180] The electronic device provides users with wireless broadband internet access through the network module 502, such as helping users send and receive emails, browse web pages, and access streaming media.

[0181] The audio output unit 503 can convert audio data received by the radio frequency unit 501 or the network module 502 or stored in the memory 509 into audio signals and output them as sound. Furthermore, the audio output unit 503 can also provide audio output related to specific functions performed by the electronic device 500 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 503 includes a speaker, a buzzer, and a receiver, etc.

[0182] Input unit 504 is used to receive audio or video signals. Input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 506. The image frames processed by GPU 5041 can be stored in memory 509 (or other storage medium) or transmitted via radio frequency unit 501 or network module 502. Microphone 5042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 501 in telephone call mode.

[0183] The electronic device 500 also includes at least one sensor 505, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 5061 according to the ambient light level, and the proximity sensor can turn off the display panel 5061 and / or backlight when the electronic device 500 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 505 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0184] The display unit 506 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0185] User input unit 507 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 507 includes a touch panel 5071 and other input devices 5072. Touch panel 5071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 5071). Touch panel 5071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 510, which receives and executes commands from the processor 510. In addition, touch panel 5071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 5071, user input unit 507 may also include other input devices 5072. Specifically, other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0186] Furthermore, the touch panel 5071 can cover the display panel 5061. When the touch panel 5071 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of touch event. Subsequently, the processor 510 provides corresponding visual output on the display panel 5061 based on the type of touch event. Although in Figure 5 In this embodiment, the touch panel 5071 and the display panel 5061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 5071 and the display panel 5061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0187] Interface unit 508 serves as an interface for connecting external devices to electronic device 500. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 508 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 500, or it can be used to transmit data between electronic device 500 and external devices.

[0188] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 509 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0189] The processor 510 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 509, and by calling data stored in the memory 509, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 510.

[0190] The electronic device 500 may also include a power supply 511 (such as a battery) for supplying power to various components. Preferably, the power supply 511 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0191] In addition, the electronic device 500 includes some functional modules not shown, which will not be described in detail here.

[0192] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0194] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0195] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0196] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0197] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0199] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0200] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0201] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A convolution operation method, characterized in that, The method relates to a neural network processor, the neural network processor including a multiplication-accumulation unit, the neural network processor including an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type, the method comprising: Obtain input data and an operator stream file; the operator stream file includes operators; Determining the operator type of the operator in the operator stream file includes: When it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, the operator type of the operator is determined to be a hardware operator; When it is determined that the operator in the operator stream file is an operator that cannot be executed on the specified hardware platform, the operator type of the operator is determined to be a software operator; When the operator type of the operator is a specified operator type, determining the convolution kernel type of the operator includes: when the operator type is a hardware operator, determining that the operator type of the operator is a specified operator type, and determining the convolution kernel type of the operator; Based on the first specified convolution kernel type and the convolution kernel type of the operator, the input data is convolved using the corresponding multiplication-accumulation unit to obtain the convolution operation result, including: When the kernel type of the operator is not the first specified kernel type, determine whether the kernel type of the operator is the second specified kernel type; When the convolution kernel type of the operator is the second specified convolution kernel type, the input channels of the input data are merged for each preset number of input channels to obtain merged input data; Determine the number of merged input channels for the merged input data; According to the preset stride, the number of merged input channels and the first specified convolution kernel type, the merged input data is subjected to convolution operation to obtain the convolution operation result; The first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset stride includes 3.

2. The method according to claim 1, characterized in that, Based on the first specified convolution kernel type and the convolution kernel type of the operator, the input data is convolved using the corresponding multiplication-accumulation unit to obtain the convolution operation result, including: When the convolution kernel type of the operator is the first specified convolution kernel type, the number of input channels of the input data is determined; According to the preset stride, the number of input channels, and the first specified convolution kernel type, the multiplication and accumulation unit is used to perform convolution operations on the input data to obtain the convolution operation result.

3. The method according to claim 1, characterized in that, After determining the operator type of the operator in the operator stream file, the method further includes: When the operator type of the operator is not the specified operator type, the preset software algorithm of the operator is invoked to perform convolution operation on the input data.

4. The method according to claim 1, characterized in that, Before acquiring the input data and operator stream file, the method further includes: An operator stream file is generated using a preset artificial intelligence compiler; the operator stream file includes a header, an operator stream, a set of hardware instructions, and a set of quantized network weights, and the operator stream includes operators; Store the operator stream file in a designated memory; When the neural network processor starts, the operator stream file is loaded into random access memory to retrieve the operator stream file from the random access memory for convolution operation.

5. The method according to claim 1, characterized in that, After performing convolution operations on the input data using the multiplication-accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator, and obtaining the convolution operation result, the method further includes: After the operator in the operator stream file has been executed once, the convolution operation result is converted into a specified format; the specified format includes at least a floating-point number format.

6. A convolution operation device, characterized in that, The device relates to a neural network processor, the neural network processor including a multiplication-accumulation unit, the neural network processor including an accelerated multiplication-accumulation unit for convolution operations of a first specified convolution kernel type, the device comprising: The data acquisition module is used to acquire input data and operator stream files; the operator stream files include operators. An operator type determination module is used to determine the operator type of the operator in the operator stream file, including: when it is determined that the operator in the operator stream file is an operator that can be executed on a specified hardware platform, determining the operator type of the operator as a hardware operator; when it is determined that the operator in the operator stream file is an operator that cannot be executed on a specified hardware platform, determining the operator type of the operator as a software operator; A kernel type determination module is used to determine the kernel type of an operator when the operator type of the operator is a specified operator type, including: when the operator type is a hardware operator, determining that the operator type of the operator is a specified operator type and determining the kernel type of the operator; The convolution operation module is used to perform convolution operations on the input data using the multiplication-accumulation unit according to the first specified convolution kernel type and the convolution kernel type of the operator, to obtain the convolution operation result, including: When the kernel type of the operator is not the first specified kernel type, determine whether the kernel type of the operator is the second specified kernel type; When the convolution kernel type of the operator is the second specified convolution kernel type, the input channels of the input data are merged for each preset number of input channels to obtain merged input data; Determine the number of merged input channels for the merged input data; According to the preset stride, the number of merged input channels and the first specified convolution kernel type, the merged input data is subjected to convolution operation to obtain the convolution operation result; The first specified convolution kernel type includes a 3*3 convolution kernel, the second specified convolution kernel type includes a 1*1 convolution kernel, and the preset stride includes 3.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-5.

Images