Instruction generation method and apparatus, pooling operation method, and electronic device

Abstract

Disclosed are an instruction generation method and device, a pooling operation method and an electronic device, and relate to the technical field of neural networks. The method comprises determining hardware parameters of a neural network accelerator performing a pooling operation; determining pooling parameters of a pooling layer performing the pooling operation; the pooling parameters comprise a first pooling step, a size of an input feature map of the pooling layer, a size of a pooling kernel, and a type of the pooling operation; and generating instructions executable by the neural network accelerator according to the pooling parameters and the hardware parameters. The technical scheme of the present disclosure considers both the hardware parameters of the neural network accelerator and the pooling parameters of the pooling layer when generating the instructions executable by the neural network accelerator, so that the neural network accelerator can support pooling of multiple steps.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to an instruction generation method and apparatus, a pooling operation method and an electronic device. Background Technology

[0002] Currently, neural networks can be modeled by simulating the neural connectivity structure of the human brain, bringing breakthroughs to large-scale data processing tasks and finding wide application in many fields such as image processing and autonomous driving. The computational process of neural networks can include pooling operations, which can reduce the size of feature maps, thereby reducing computational load. Summary of the Invention

[0003] Existing neural network accelerators only support pooling with a fixed stride (e.g., pooling with a stride of 2), and may not natively support pooling with other strides.

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an instruction generation method and apparatus, a pooling operation method, and an electronic device. When generating instructions executable by a neural network accelerator, both the hardware parameters of the neural network accelerator and the pooling parameters of the pooling layer are considered, thereby enabling the neural network accelerator to support pooling with multiple step sizes.

[0005] According to one aspect of this application, a method for generating instructions for pooling operations in a neural network accelerator is provided, comprising: first, determining the hardware parameters of the neural network accelerator performing the pooling operation; second, determining the pooling parameters of the pooling layer performing the pooling operation, the pooling parameters including a first pooling step size, the size of the input feature map of the pooling layer, the size of the pooling kernel, and the type of pooling operation; and finally, generating instructions executable by the neural network accelerator based on the pooling parameters and the hardware parameters.

[0006] Based on this scheme, executable instructions for the neural network accelerator are generated by considering the hardware parameters of the neural network accelerator performing pooling operations and the pooling parameters of the pooling layers. In other words, when generating executable instructions for the neural network accelerator, both the hardware parameters of the neural network accelerator and the pooling parameters of the pooling layers are taken into account, thus enabling the neural network accelerator to support pooling with various step sizes.

[0007] According to one aspect of this application, a method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator is provided, comprising: first, determining the pooling parameters of the pooling layers in the neural network model that the neural network accelerator needs to execute, the pooling parameters including a first pooling step size, the type of pooling operation of the pooling layer, and the pooling kernel size corresponding to the pooling layer; then, processing the input feature map of the pooling layer based on the pooling parameters to obtain the pooled output feature map.

[0008] Based on this scheme, by determining the pooling parameters of the pooling layer in the neural network model that the neural network accelerator needs to execute, and processing the input feature map of the pooling layer according to the pooling parameters to obtain the pooled output feature map, the neural network accelerator can perform the pooling operation required by the pooling layer using the existing hardware parameters without changing the hardware parameters.

[0009] According to one aspect of this application, an instruction generation apparatus for performing pooling operations on a neural network accelerator is provided, comprising: a first determining module for determining hardware parameters of the neural network accelerator performing the pooling operation; a second determining module for determining pooling parameters of a pooling layer performing the pooling operation; the pooling parameters including a first pooling step size, the size of the input feature map of the pooling layer, the size of the pooling kernel, and the type of pooling operation; and an instruction generation module for generating executable instructions for the neural network accelerator based on the pooling parameters and the hardware parameters.

[0010] According to one aspect of this application, a computer-readable storage medium is provided, the storage medium storing a computer program for performing any of the methods described above.

[0011] According to one aspect of this application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement any of the methods described above. Attached Figure Description

[0012] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is an exemplary structural diagram of the system to which this application applies.

[0014] Figure 2This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application.

[0015] Figure 3 This is a schematic diagram of the structure of another electronic device provided in an exemplary embodiment of this application.

[0016] Figure 4 This is a flowchart illustrating an instruction generation method for pooling operations in a neural network accelerator, provided by an exemplary embodiment of this application.

[0017] Figure 5 This is a schematic diagram of an input feature map provided in an exemplary embodiment of this application.

[0018] Figure 6 This is a flowchart illustrating another instruction generation method for pooling operations in a neural network accelerator, provided by an exemplary embodiment of this application.

[0019] Figure 7 This is a schematic diagram of a pooling operation provided in an exemplary embodiment of this application.

[0020] Figure 8 This is a schematic diagram of an enlarged feature map provided in an exemplary embodiment of this application.

[0021] Figure 9 This is a schematic diagram of another pooling operation provided by an exemplary embodiment of this application.

[0022] Figure 10 This is a schematic diagram of another pooling operation provided in an exemplary embodiment of this application.

[0023] Figure 11 This is a flowchart illustrating a method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator, provided by an exemplary embodiment of this application.

[0024] Figure 12 This is a flowchart illustrating another method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator, provided by an exemplary embodiment of this application.

[0025] Figure 13 This is a flowchart illustrating another method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator, provided by an exemplary embodiment of this application.

[0026] Figure 14 This is a structural diagram of an instruction generation apparatus provided in an exemplary embodiment of this application.

[0027] Figure 15 This is a structural diagram of a pooling computing device provided in an exemplary embodiment of this application.

[0028] Figure 16 This is a structural diagram of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0029] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0030] The most commonly used pooling method in neural networks is with a stride of 2, and typically, neural network accelerators support pooling with a stride of 2. However, with the development of neural network algorithms, pooling with other strides (e.g., pooling with a stride of 1) has emerged. However, neural network accelerators may not natively support these other stride pooling methods. For example, the range of hardware parameters that a brain processing unit (BPU) can support is related to its hardware overhead. The wider the range that the BPU can support, the greater the hardware overhead. Therefore, with limited hardware overhead, the range of hardware parameters that the BPU can support is limited.

[0031] To address the limitation of step size supported by the hardware parameters of neural network accelerators, this application provides an instruction generation method for pooling operations in neural network accelerators. This method considers both the hardware parameters of the neural network accelerator and the pooling parameters of the pooling layer when generating executable instructions for the neural network accelerator. As a result, the generated executable instructions not only meet the computational requirements of the pooling layer, but also that the hardware of the neural network accelerator can support the computational requirements of the pooling layer, enabling pooling operations with multiple step sizes.

[0032] Figure 1 The present application provides a system for performing neural network operations. The system includes a compilation-side device 10 and a runtime-side device 11. The compilation-side device is used to generate a sequence of instructions for the neural network, and the runtime-side device 11 is used to perform neural network operations by running the sequence of instructions provided by the compilation-side device 10. These operations include, but are not limited to, convolution operations, pooling operations, activation function operations, etc.

[0033] In some embodiments, the compiler-side device 10 may be implemented by an electronic device 20. The electronic device 20 may include, but is not limited to, a personal computer, a server computer, a multiprocessor system, a microprocessor-based system, etc.

[0034] In some embodiments, the operation-side device 11 may be implemented by an electronic device 30. The electronic device 30 may include, but is not limited to, a neural network accelerator or a neural network chip.

[0035] Figure 2 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application, as shown below. Figure 2 As shown, the electronic device 20 includes at least one processor 21 and a memory 22.

[0036] The processor 21 is used to execute the instruction generation method for pooling operations in a neural network accelerator according to the embodiments of this application. The processor 21 can be a high-performance central processing unit (CPU), graphics processing unit (GPU), or other processor that supports the compilation of neural network instruction sequences, or other processing units with compilation processing capability and / or instruction execution capability.

[0037] The memory 22 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random-access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 21 may execute the program instructions to implement the instruction generation method for pooling operations in a neural network accelerator described below.

[0038] In some embodiments, such as Figure 2 As shown, the electronic device 20 may also include an input device 23 and an output device 24. The input device 23 may include, for example, a keyboard, mouse, etc., for compilers to input parameters of the neural network, such as the size of the input feature map of each layer, the size of the output feature map, and the convolution kernel. The output device 24 may include, for example, a display, speaker, printer, and communication networks and their connected remote output devices, etc., for compilers to view the final compilation results or intermediate data during the compilation process.

[0039] It should be noted that the embodiments of this application do not limit the specific structure of the electronic device 20, and the electronic device 20 may include more than Figure 2 The number of components shown may be more or less. For example, electronic device 20 may also include components such as buses and input / output interfaces.

[0040] Figure 3 This is a schematic diagram of the structure of an electronic device 30 provided in an embodiment of this application, as shown below. Figure 3As shown, the electronic device 30 may include a control unit 31 and a computing unit 32. The control unit 31 is used to run instructions to control the computing unit 32 to perform the following method for pooling operations on the pooling layers in the neural network model.

[0041] The control unit 31 may include one or more processors or other processing units with neural network computing capabilities and / or instruction execution capabilities. The processor may include, but is not limited to, GPUs, brain processing units (BPUs), tensor processing units (TPUs), and other processors that support neural network-related computations. In some embodiments, the control unit 31 is used to control the computing unit 32 to perform corresponding operations according to instructions executable by the neural network accelerator (e.g., convolution operation instructions, average calculation instructions, feature map amplification instructions, and pooling operation instructions).

[0042] The computing unit 32 includes an arithmetic circuit that can be configured to perform neural network operations, including but not limited to any one or more of operations such as convolution, pooling, and activation function operations.

[0043] In some embodiments, the electronic device 30 may further include a buffer memory 33 and an off-chip memory 34. The buffer memory 33 may include one or more independent cache memories or a processing unit with data caching capabilities, and may access the off-chip memory 34 under the control of the control unit 31. In some embodiments, the buffer memory 31 may be a fast, small-capacity static access memory, such as static random access memory (SRAM). Exemplarily, the buffer memory 33 may be configured to receive at least a portion of the input feature data and / or the data of the convolution kernel from the off-chip memory 34 and cache it for use by the computing unit 32, and may also be configured to receive and cache the output feature data output by the computing unit 32, and output the output feature data to the off-chip memory 34.

[0044] The off-chip memory 34 may include one or more independent memories or processing units with data storage capabilities, and may be accessed by the buffer memory 33 under the control of the control unit 31. In some embodiments, the off-chip memory 34 is a slow, large-capacity dynamic access memory, such as dynamic random access memory (DRAM). In some embodiments, the off-chip memory 34 is double data rate (DDR).

[0045] It should be noted that the embodiments of this application do not limit the specific structure of the electronic device 30, and the electronic device 30 may include more than Figure 3 The number of components shown may be more or less. For example, electronic device 30 may also include components such as buses.

[0046] The methods described in the following embodiments of this application can be applied to many fields that require neural network computation, such as image processing and intelligent driving. For example, image data can be processed in the field of image processing by executing the methods provided in the embodiments of this application.

[0047] Figure 4 This application provides an instruction generation method for pooling operations in a neural network accelerator. This method can be applied to the compilation-side device 10, such as... Figure 4 As shown, the method includes steps 401-403.

[0048] Step 401: Determine the hardware parameters of the neural network accelerator that will perform the pooling operation.

[0049] For example, the neural network accelerator can be any accelerator capable of executing neural network algorithms, such as a brain processing unit (BPU), a neural network processing unit (NPU), etc. This disclosure does not limit the type of neural network accelerator; the following embodiments use a BPU as an example for illustrative purposes.

[0050] The hardware parameters of a neural network accelerator include the hardware parameters of each layer supported by the accelerator, such as convolutional layers, pooling layers, and activation function layers. For example, the hardware parameters of a neural network accelerator include, but are not limited to, the pooling stride (also known as the second pooling stride) and the convolution stride supported by the accelerator.

[0051] Step 402: Determine the pooling parameters of the pooling layer to be used for pooling operations.

[0052] The pooling parameters of this pooling layer include the first pooling stride, the size of the input feature map of the pooling layer, the size of the pooling kernel, and the type of pooling operation. This first pooling stride differs from the second pooling stride supported by neural network accelerators.

[0053] The first pooling step size is the movement step size of the pooling kernel when performing pooling operations on the input feature map of the pooling layer. The first pooling step size can include a horizontal pooling step size and a vertical pooling step size. The horizontal pooling step size and the vertical pooling step size can be the same or different, and this application embodiment does not limit this. The following embodiment uses the example where the horizontal pooling step size and the vertical pooling step size are the same and both are 1 for illustrative purposes.

[0054] The dimensions of the pooling core include width and height. This application does not limit the size of the pooling core; the following embodiments use a 3x3 pooling core as an example for illustrative purposes.

[0055] The input feature map for a pooling layer can be the output feature map of a convolutional layer, the output feature map of other layers in a neural network, or a feature map that has not been processed by other layers in the neural network. The size of the input feature map for this pooling layer includes both width and height. For example, Figure 5 As shown, the input feature map of the pooling layer has a size of 5*6, a height of 5, and a width of 6. This input feature map can include the pixel values ​​of 30 pixels.

[0056] Pooling operations include average pooling and max pooling. Average pooling is the average value of the matrix elements of the input feature map enclosed by the pooling kernel; max pooling is the maximum value of the matrix elements of the feature map enclosed by the pooling kernel.

[0057] It should be noted that the embodiments of this application do not limit the moving order of the pooling kernel and the convolution kernel. For example, the pooling kernel can move sequentially from left to right and from top to bottom according to the pooling stride, or it can move sequentially from top to bottom and from left to right according to the pooling stride, etc., as long as all output points can be calculated. The following embodiments use the example of the pooling kernel moving sequentially from left to right and from top to bottom according to the pooling stride, and the convolution kernel moving sequentially from left to right and from top to bottom according to the convolution stride, to illustrate the concept.

[0058] For example, taking average pooling as the pooling operation type, with an input feature map size of 5*6, a pooling kernel size of 3*3, and a first pooling stride of 1, as an example... Figure 5As shown, the first set of matrix elements enclosed by the pooling kernel includes X11, X12, X13, X21, X22, X23, X31, X32, and X33. The average value of these matrix elements is then taken to obtain the average pooled output value Z1 = (X11 + X12 + X13 + X21 + X22 + X23 + X31 + X32 + X33) / 9. Then, following a first pooling step size of 1, the pooling kernel is moved sequentially from left to right and from top to bottom, and the average value of each set of matrix elements enclosed by the pooling kernel is taken. Finally, the pooled output feature map is obtained, with a size of 3*4.

[0059] For example, taking max pooling as the pooling operation type, with an input feature map size of 5*6, a pooling kernel size of 3*3, and a first pooling stride of 1, as follows: Figure 5 As shown, the first set of matrix elements enclosed by the pooling kernel includes X11, X12, X13, X21, X22, X23, X31, X32, and X33. The maximum value among these elements is then taken to obtain the maximum pooled output value for that element. Next, the pooling kernel is moved sequentially from left to right and from top to bottom, with a first pooling step size of 1, and the maximum value is taken for each set of matrix elements enclosed by the kernel. Finally, the pooled output feature map is obtained, with a size of 3*4.

[0060] Step 403: Generate executable instructions for the neural network accelerator based on the pooling parameters and hardware parameters.

[0061] In this embodiment, the compiler-side device fully considers the hardware parameters of the neural network accelerator and the pooling parameters of the pooling layer when generating executable instructions for the neural network accelerator. Therefore, the generated executable instructions not only meet the computational requirements of the pooling layer, but the hardware of the neural network accelerator can also support the computational requirements of the pooling layer, thereby enabling pooling operations with multiple step sizes. In other words, this application allows the neural network accelerator to support pooling operations with both a second and a first pooling step size without changing the hardware parameters of the neural network accelerator.

[0062] In some embodiments, such as Figure 6 As shown, step 403 above may include steps 4031-4032.

[0063] Step 4031: Based on the type of pooling operation, determine the support relationship between the first pooling step size and the hardware parameters.

[0064] When the pooling operation type of the pooling layer is max pooling, the compiler can determine whether the hardware parameters of the neural network accelerator support pooling operations with a first pooling step. If the compiler determines that the hardware parameters of the neural network accelerator support pooling operations with a first pooling step (i.e., the pooling step supported by the neural network accelerator is the same as the first pooling step), then the compiler can directly generate pooling operation instructions. If the compiler determines that the hardware parameters of the neural network accelerator do not support pooling operations with a first pooling step (i.e., the pooling step supported by the neural network accelerator is different from the first pooling step), then the compiler can generate executable instructions for the neural network accelerator based on this support relationship.

[0065] For example, if the first pooling step size is 1 and the second pooling step size supported by the BPU is 2, and the pooling operation type of the pooling layer is max pooling, then the compilation-side device determines the support relationship between the first pooling step size and the hardware parameters based on the pooling type as follows: the BPU does not support pooling operations with a step size of the first pooling step size, or the second pooling step size supported by the BPU is different from the first pooling step size.

[0066] When the pooling operation type of the pooling layer is average pooling, if the compilation device determines that the hardware parameters of the neural network accelerator do not support pooling operations with the first pooling stride, then the compilation device further determines whether the hardware parameters of the neural network accelerator support convolution operations with the same stride as the first pooling stride. If the compilation device determines that the hardware parameters of the neural network accelerator support convolution operations with the same stride as the first pooling stride (i.e., the convolution stride supported by the hardware parameters of the neural network accelerator is the same as the first pooling stride), then the compilation device can generate executable instructions for the neural network accelerator based on this support relationship.

[0067] For example, taking a first pooling stride of 1, a second pooling stride supported by the BPU of 2, and a convolution stride supported by the BPU of 1 as an example, if the pooling operation type of the pooling layer is average pooling, then the compilation-side device determines the support relationship between the first pooling stride and the hardware parameters based on the pooling type as follows: the BPU does not support pooling operations with the first pooling stride (which can also be said that the second pooling stride supported by the BPU is different from the first pooling stride), and the BPU supports convolution operations with the same stride as the first pooling stride (which can also be said that the convolution stride supported by the BPU is the same as the first pooling stride).

[0068] Step 4032: Based on the support relationship, generate executable instructions for the neural network accelerator.

[0069] When the pooling operation type is average pooling, and the support relationship is: the hardware parameters of the neural network accelerator do not support pooling operations with a first pooling stride (i.e., the second pooling stride supported by the neural network accelerator is different from the first pooling stride), and the hardware parameters of the neural network accelerator support convolution operations with the same stride as the first pooling stride (i.e., the convolution stride supported by the neural network accelerator is the same as the first pooling stride), the above step 4032 includes: the compilation-side device determines a convolution kernel and a first value with the same pooling kernel size based on the support relationship and the pooling kernel size; and then generates convolution operation instructions and average value calculation instructions executable by the neural network accelerator based on the convolution kernel and the first value.

[0070] For example, taking a pooling kernel size of 3*3 as an example, the size of the convolution kernel determined by the pooling kernel is also 3*3, and the values ​​of all elements in the convolution kernel are the same, which is the second value. This application embodiment does not limit the specific value of the second value; the second value can be an integer, a decimal, etc.

[0071] The first value is determined based on the second value and the size of the convolution kernel. Taking the second value as 'a' and the kernel size as 'm*n' as an example, then the first value 'b' = a*m*n. For ease of understanding, the following example uses the second value as 1, the kernel size as 3*3, and the first value as 9 for illustrative purposes.

[0072] The convolution operation instruction is used to perform a convolution operation between the matrix elements participating in the pooling operation in the input feature map and the convolution kernel to obtain the convolution output value of the matrix element. The average calculation instruction is used to divide the convolution output value of the matrix element by a first value to obtain the average pooling output value of the matrix element.

[0073] For example, such as Figure 7As shown, taking average pooling as the pooling operation type, the input feature map size of the pooling layer is 5*6, the pooling kernel size is 3*3, the first pooling stride is 1, and the value of all elements in the pooling kernel is 1. Taking the matrix elements participating in the pooling operation in the input feature map as X11, X12, X13, X21, X22, X23, X31, X32, and X33 as an example, the convolution operation instruction is used to perform a convolution operation on these matrix elements with the convolution kernel to obtain the convolution output value Y1 = X11*1 + X12*1 + X13*1 + X21*1 + X22*1 + X23*1 + X31*1 + X32*1 + X33*1. The average value calculation instruction is used to divide the convolution output value Y1 of the matrix element by the first value 9 to obtain the average pooling output value Z1 = Y1 / 9 of the matrix element. Clearly, the result of average pooling on matrix elements X11, X12, X13, X21, X22, X23, X31, X32, X33 is the same as the result of convolving matrix elements X11, X12, X13, X21, X22, X23, X31, X32, X33 first and then averaging them.

[0074] Understandably, when the pooling operation type of the pooling layer is average pooling, if the second pooling stride supported by the neural network accelerator is different from the first pooling stride, and the hardware parameters of the neural network accelerator support convolution operations with the same stride as the first pooling stride, then convolution operations can be used instead of pooling operations. Furthermore, to ensure that the result of the convolution operation is equivalent to the result of the pooling operation, the convolution output value can be divided by the first value after the convolution operation. Using this method, the executable instructions of the neural network accelerator can be generated based on the support relationship between the hardware parameters of the neural network accelerator and the first pooling stride. Therefore, without changing the hardware parameters of the neural network accelerator, it is possible to enable the neural network accelerator to support pooling operations with both the first and second pooling strides.

[0075] When the pooling operation type is max pooling and the support relationship is that the second pooling step size supported by the neural network accelerator is different from the first pooling step size, the above step 4032 includes: the compilation side device determines the amplified pooling kernel size based on the support relationship and the size of the pooling kernel; and generates feature map amplification instructions and pooling operation instructions executable by the neural network accelerator based on the amplified pooling kernel size.

[0076] The hardware parameters of the neural network accelerator support a second pooling step size. The feature map magnification instruction is used to magnify the input feature map using a bilinear interpolation algorithm, based on the first and second pooling step sizes, to obtain a magnified feature map. The pooling operation instruction includes a second pooling step size and is used to perform pooling operations on the matrix elements participating in the pooling operation in the magnified feature map based on the magnified pooling kernel size, to obtain the maximum pooling output value of the matrix elements.

[0077] For example, such as Figure 8 As shown, taking max pooling as the pooling operation type, with an input feature map size of 5*6 and a pooling kernel size of 3*3, a first pooling stride of 1, and a second pooling stride of 2 as an example, the feature map magnification instruction is used to magnify the input feature map by a factor of 1 using bilinear interpolation, based on the size of the first pooling stride 1 and the second pooling stride 2, resulting in a magnified feature map with a size of 9*11. Furthermore, the values ​​of the interpolation points in the magnified feature map will not exceed the maximum values ​​of the input points in the input feature map. For example, the value of interpolation point a1 is the average of X11 and X12, so the value of a1 will not exceed the maximum value of X11 and X12. Similarly, the value of interpolation point f1 is the average of X11 and X21, so the value of f1 will not exceed the maximum value of X11 and X21. And again, the value of interpolation point f2 is the average of a1 and b1, so the value of f2 will not exceed the maximum value of a1 and b1.

[0078] like Figure 9 As shown, based on the size of the pooling kernel (3*3), the size of the first pooling step (1), and the size of the second pooling step (2), the size of the enlarged pooling kernel can be determined to be 5*5. The pooling operation instruction is used to perform max pooling on the matrix elements (e.g., X11, a1, X12, a2, X13, f1, f2, f3, f4, f5, X21, b1, X22, b2, X23, g1, g2, g3, g4, g5, X31, c1, X32, c2, X33) participating in the pooling operation in the enlarged feature map, based on the enlarged pooling kernel size of 5*5, to obtain the max pooling output value Y1' of the matrix elements. Since the elements a1, a2, f1, f2, f3, f4, f5, b1, b2, g1, g2, g3, g4, g5, c1, and c2 in this matrix are all interpolation points, and the values ​​of these interpolation points will not exceed the values ​​of X11, X12, X13, X21, X22, X23, X31, X32, and X33 in the input feature map, the maximum value of the matrix elements participating in the pooling operation in the amplified feature map is obtained by using a 5*5 pooling kernel, which yields the same maximum value as the maximum value of the matrix elements participating in the pooling operation in the input feature map obtained by using a 3*3 pooling kernel.

[0079] like Figure 10As shown, the second pooling step size 2 moves to the right. The pooling operation instruction is used to perform max pooling operation on the matrix elements (e.g., X12, a2, X13, a3, X14, f3, f4, f5, f6, f7, X22, b2, X23, b3, X24, g3, g4, g5, g6, g7, X32, c2, X33, c3, X34) in the magnified feature map based on the magnified pooling kernel size of 5*5, so as to obtain the max pooling output value Y2' of the matrix elements. Since the elements a2, a3, f3, f4, f5, f6, f7, b2, b3, g3, g4, g5, g6, g7, c2, and c3 in this matrix are all interpolation points, and the values ​​of these interpolation points will not exceed the values ​​of X12, X13, X14, X22, X23, X24, X32, X33, and X34 in the input feature map, the maximum value of the matrix elements participating in the pooling operation in the amplified feature map is obtained by using a 5*5 pooling kernel, which yields the same maximum value as the maximum value of the matrix elements participating in the pooling operation in the input feature map obtained by using a 3*3 pooling kernel.

[0080] Understandably, when the pooling operation type of the pooling layer is max pooling, if the second pooling stride supported by the neural network accelerator is different from the first pooling stride, and the first pooling stride is smaller than the second pooling stride supported by the neural network accelerator, then the input feature map can be enlarged, and pooling operations can be performed on the enlarged feature map based on the second pooling stride supported by the neural network accelerator. Using this method, instructions executable by the neural network accelerator can be generated based on the support relationship between the hardware parameters of the neural network accelerator and the first pooling stride. Therefore, without changing the hardware parameters of the neural network accelerator, it is possible to enable the neural network accelerator to implement the pooling operation of the first pooling stride using its supported second pooling stride.

[0081] Figure 11 This application provides a method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator. This method can be applied to the runtime device 11, such as... Figure 11 As shown, the method includes steps 1101-1102.

[0082] Step 1101: Determine the pooling parameters of the pooling layers in the neural network model that the neural network accelerator needs to execute.

[0083] The pooling parameters of this pooling layer include the first pooling step size, the type of pooling operation, and the corresponding pooling kernel size. Pooling operation types include, but are not limited to, average pooling and max pooling. For detailed explanations of the pooling parameters, please refer to the foregoing embodiments, which will not be repeated here. The following embodiments use a first pooling step size of 1 and a corresponding pooling kernel size of 3*3 as examples for illustrative purposes.

[0084] For example, taking a neural network accelerator as a BPU, we can determine the parameters such as the pooling step size, pooling operation type, and pooling kernel size of the pooling layer in the neural network model that the BPU needs to execute.

[0085] Step 1102: Based on the pooling parameters, process the input feature map of the pooling layer to obtain the pooled output feature map.

[0086] The method for performing pooling operations on pooling layers in a neural network model provided in this application embodiment determines the pooling parameters of the pooling layers that the neural network accelerator needs to execute in the neural network model, and processes the input feature map of the pooling layer according to the pooling parameters to obtain the pooled output feature map. This enables the neural network accelerator to perform the pooling operations required by the pooling layer using existing hardware parameters without changing the hardware parameters.

[0087] When the pooling stride of the pooling layer differs from the pooling stride supported by the neural network accelerator, the neural network accelerator can process the input feature map of the pooling layer accordingly based on the different pooling operation types to obtain the pooled output feature map. The specific implementation of step 1102 will be explained in detail below, taking average pooling and max pooling as examples.

[0088] In some embodiments, if the pooling operation type of the pooling layer is average pooling, such as Figure 12 As shown, step 1102 above may include steps 1201-1204.

[0089] Step 1201: Determine the convolution stride supported by the neural network accelerator.

[0090] For example, taking a neural network accelerator (BPU) as an example, the convolution stride supported by the BPU can be the same as or different from the first pooling stride of the pooling layer. This embodiment of the application illustrates this by taking an example where the pooling stride supported by the BPU is different from the first pooling stride of the pooling layer, but the convolution stride supported by the BPU is the same as the first pooling stride. For example, the first pooling stride is 1, the convolution stride supported by the BPU is also 1, and the pooling stride supported by the BPU is 2.

[0091] Step 1202: Determine the convolution kernel and the first value based on the convolution stride and pooling kernel size.

[0092] The convolution kernel can be determined based on the pooling kernel. The size of the convolution kernel is the same as that of the pooling kernel, and all parameters in the convolution kernel have the same value.

[0093] For example, taking a pooling kernel size of 3*3 as an example, the size of the convolution kernel can be determined to be 3*3 based on this pooling kernel, and the values ​​of all parameters in this convolution kernel are all second values. These second values ​​can be integers or decimals, and this application does not limit them. For ease of explanation, the following embodiments use the example of all parameters in the convolution kernel having a value of 1 for illustrative purposes.

[0094] The first value is determined by the second value and the size of the convolution kernel. For example, if all parameters in the convolution kernel are the second value, and the second value is 'a', and the size of the convolution kernel is m*n, then the first value b = a*m*n.

[0095] For example, if the kernel size is 3x3 and all parameters in the kernel are 1, then the first value is 9. As another example, if the kernel size is 3x3 and all parameters in the kernel are 2, then the first value is 18. And as yet another example, if the kernel size is 4x4 and all parameters in the kernel are 2, then the first value is 32.

[0096] Step 1203: Perform convolution operations on the input feature maps of the convolution kernel and pooling layer to obtain the convolutional feature map.

[0097] like Figure 7 As shown, taking a convolution stride of 1, a kernel size of 3*3, and an input feature map size of 5*6 for the pooling layer as an example, the BPU can scan the input feature map of the pooling layer sequentially through the data window with a convolution stride of 1, from left to right and from top to bottom. The pooling matrix elements in the input feature map obtained from the first scan include X11, X12, X13, X21, X22, X23, X31, X32, and X33. The pooling matrix elements in the input feature map obtained from the second scan include X12, X13, X14, X22, X23, X24, X32, X33, and X34. The pooling matrix elements in the input feature map obtained from the third scan include X13, X14, X15, X23, X24, X25, X33, X34, and X35, and so on.

[0098] For example, the BPU sequentially scans the input feature map of the pooling layer through the data window to determine the matrix elements for pooling the input feature map of the pooling layer, and performs a convolution operation on the matrix elements scanned by each data window with the convolution kernel to obtain a convolution output value.

[0099] For example, such as Figure 7As shown, taking a 3*3 matrix with all values ​​of 1 as the convolution kernel as an example, the matrix elements obtained in the first scan include X11, X12, X13, X21, X22, X23, X31, X32, and X33. Performing a convolution operation between this matrix element and the convolution kernel yields the convolution output value Y1 = X11*1 + X12*1 + X13*1 + X21*1 + X22*1 + X23*1 + X31*1 + X32*1 + X33*1. The matrix elements obtained from the second scan include X12, X13, X14, X22, X23, X24, X32, X33, and X34. Performing a convolution operation on these matrix elements with the convolution kernel yields the convolution output value Y2 = X12*1 + X13*1 + X14*1 + X22*1 + X23*1 + X24*1 + X32*1 + X33*1 + X34*1. This process is repeated to obtain Y3 through Y12. Figure 7 As shown, after performing a convolution operation between the input feature map of the pooling layer and the convolution kernel, a convolutional feature map can be obtained, which has a size of 3*4.

[0100] Step 1204: Divide the parameters in the convolutional feature map by the first value to obtain the pooled output feature map.

[0101] Understandably, when the pooling operation type of the pooling layer is average pooling, if the pooling stride supported by the neural network accelerator is different from the pooling stride of the pooling layer, but the convolution stride supported by the neural network accelerator is the same as the pooling stride of the pooling layer, then convolution can be used instead of average pooling. When using convolution instead of average pooling, it is necessary to ensure that the values ​​of all parameters in the convolution kernel are the same, and in order to ensure that the output feature map is exactly the same as that of average pooling, the feature map obtained by convolution needs to be further processed.

[0102] For example, after obtaining the convolution output value in step 1203, in order to ensure that the output feature map is exactly the same as that obtained by average pooling when using convolution instead of average pooling, it is also necessary to divide each convolution output value by a first value.

[0103] For example, such as Figure 7As shown, taking the matrix elements scanned in the first scan as X11, X12, X13, X21, X22, X23, X31, X32, X33, with the first value being 9, step 1203 performs a convolution operation on these matrix elements with the convolution kernel to obtain the convolution output value Y1. Step 1204 divides the convolution output value Y1 obtained in step 1203 by 9 to obtain Z1 = Y1 / 9. That is, Z1 = (X11*1 + X12*1 + X13*1 + X21*1 + X22*1 + X23*1 + X31*1 + X32*1 + X33*1) / 9. The output data Z' after performing average pooling on these matrix elements is (X11 + X12 + X13 + X21 + X22 + X23 + X31 + X32 + X33) / 9. Clearly, the average pooling output value Z' is the same as Z1.

[0104] For example, such as Figure 7 As shown, taking the matrix elements obtained in the second scan as X12, X13, X14, X22, X23, X24, X32, X33, X34, with the first value being 9, step 1203 performs a convolution operation on these matrix elements with the convolution kernel to obtain the convolution output value Y2. Step 1204 divides the convolution output value Y2 obtained in step 1203 by 9 to obtain Z2 = Y2 / 9. That is, Z2 = (X12*1 + X13*1 + X14*1 + X22*1 + X23*1 + X24*1 + X32*1 + X33*1 + X34*1) / 9. The output data Z” after performing average pooling on the matrix elements is (X12+X13+X14+X22+X23+X24+X32+X33+X34) / 9. Obviously, the average pooling output value Z” is the same as Z2.

[0105] like Figure 7 As shown, by dividing each matrix element in the convolutional feature map obtained in step 1203 by the first value, the output feature map of the pooling layer can be obtained. This output feature map is the same as the output feature map obtained by average pooling the input feature map using the first pooling stride.

[0106] The method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator, provided in this application embodiment, allows for the use of convolution instead of average pooling when the pooling operation type of the pooling layer is average pooling. If the convolution stride supported by the neural network accelerator is the same as the pooling stride of the pooling layer, then convolution can be used instead of average pooling. When using convolution instead of average pooling, it is necessary to ensure that the values ​​of all parameters in the convolution kernel are the same, and the output value after convolution must be processed to obtain an output feature map that is exactly the same as that of average pooling. This application allows the neural network accelerator to support pooling operations with both a first and a second pooling stride without changing the hardware parameters of the accelerator.

[0107] In some embodiments, if the pooling operation type of the pooling layer is max pooling, such as Figure 13 As shown, step 1102 above includes steps 1301-1305.

[0108] Step 1301: Determine the second pooling step size supported by the neural network accelerator.

[0109] The second pooling step size is larger than the first pooling step size of the pooling layer. This application does not limit the value of the second pooling step size supported by the neural network accelerator. The following embodiments use a second pooling step size of 2 supported by the neural network accelerator and a first pooling step size of 1 as an example for illustrative purposes.

[0110] Step 1302: Determine the amplification factor based on the first pooling step size and the second pooling step size.

[0111] The magnification factor is determined by the first pooling step size and the second pooling step size. For example, the magnification factor is related to the numerical values ​​of the first and second pooling step sizes; for instance, the magnification factor is a multiple of the second pooling step size compared to the first pooling step size.

[0112] For example, taking the second pooling step size supported by BPU as 2 and the first pooling step size of the pooling layer as 1, the amplification factor can be determined based on the numerical values ​​of the first and second pooling step sizes, and the amplification factor can be 2.

[0113] Step 1303: Based on the magnification factor, the input feature map is magnified using a bilinear interpolation algorithm to obtain the magnified feature map.

[0114] For example, when upscaling the input feature map of the pooling layer according to the magnification factor, interpolation points can be inserted into the input feature map of the pooling layer. Moreover, in order to obtain an output feature map that is exactly the same as that of max pooling, the value of the interpolation point inserted into the input feature map does not exceed the maximum value of the input point in the input feature map.

[0115] For example, such as Figure 8 As shown, taking a second pooling stride of 2 supported by the BPU and a first pooling stride of 1 as an example, based on the first and second pooling strides, the 5*5 input feature map can be magnified by a factor of 2 to obtain a magnified feature map with a size of 9*11. Furthermore, the value of interpolation point a1 in the magnified feature map does not exceed the larger of X11 and X12. Similarly, the value of interpolation point f1 does not exceed the larger of X11 and X21. And again, the value of interpolation point f2 does not exceed the larger of a1 and b1.

[0116] Step 1304: Determine the amplified pooling kernel size based on the amplification factor and the pooling kernel size.

[0117] For example, taking a pooling kernel size of 3*3 and a magnification factor of 2 as an example, the magnified pooling kernel size can be determined to be 5*5 based on the magnification factor and the pooling kernel size. Moreover, the size of the output feature map obtained by performing pooling operation on the input feature map of the pooling layer using the pooling kernel is the same as the size of the output feature map obtained by performing pooling operation on the magnified input feature map using the magnified pooling kernel.

[0118] For example, such as Figure 8 As shown, taking an input feature map of size 5*6, a first pooling stride of 1, and a second pooling stride of 2 supported by the BPU as an example, if a 3*3 pooling kernel is used with a first pooling stride of 1 to perform max pooling on the 5*6 input feature map, a 3*4 output feature map can be obtained. If a 5*5 enlarged pooling kernel is used with a second pooling stride of 2 to perform max pooling on the 9*11 enlarged feature map, a 3*4 output feature map can also be obtained.

[0119] Step 1305: Based on the second pooling step size and the enlarged pooling kernel size, perform pooling operations on the enlarged feature map to obtain the pooled output feature map.

[0120] For example, such as Figure 9As shown, taking a second pooling step size of 2 supported by the BPU, a magnified feature map size of 9*11, and a magnified pooling kernel size of 5*5 as an example, the BPU determines that the first set of matrix elements for pooling operations in the magnified feature map includes X11, a1, X12, a2, X13, f1, f2, f3, f4, f5, X21, b1, X22, b2, X23, g1, g2, g3, g4, g5, X31, c1, X32, c2, X33. Performing max pooling on the first set of matrix elements yields the max pooling output value Y1' of that first set of matrix elements. Since the values ​​of interpolation points a1, b1, and c1 in the first set of matrix elements will not exceed the larger of X11 and X12 in the input feature map, respectively; the values ​​of interpolation points a2, b2, and c2 in the input feature map will not exceed the larger of X12 and X13 in the input feature map; the values ​​of interpolation points b1, b2, and c2 in the input feature map will not exceed the larger of X22 and X23 in the input feature map; the values ​​of interpolation points c1, c2, and f1 in the input feature map will not exceed the larger of X11 and X21 in the input feature map; the values ​​of interpolation points f3 and f3 in the input feature map will not exceed the larger of X12 and X22 in the input feature map; and the values ​​of interpolation points f5 and f6 in the input feature map will not exceed the larger of X11 and X21 in the input feature map. The larger value among X13 and X23 in the feature map, the value of interpolation point f2 will not exceed the larger value among interpolation points a1 and b1, and the value of interpolation point f4 will not exceed the larger value among interpolation points a2 and b2; the value of interpolation point g1 will not exceed the larger value among X21 and X31 in the input feature map, the value of interpolation point g3 will not exceed the larger value among X22 and X32 in the input feature map, the value of interpolation point g5 will not exceed the larger value among X23 and X33 in the input feature map, the value of interpolation point g2 will not exceed the larger value among interpolation points b1 and c1, and the value of interpolation point g4 will not exceed the larger value among interpolation points b2 and c2. Therefore, the result of finding the maximum value of the first group of matrix elements is the same as the result of finding the maximum value of X11, X12, X13, X21, X22, X23, X31, X32, and X33.

[0121] For example, such as Figure 10As shown, the BPU moves 2 to the right with the second pooling step size to determine the second set of matrix elements for pooling operations in the magnified feature map, including X12, a2, X13, a3, X14, f3, f4, f5, f6, f7, X22, b2, X23, b3, X24, g3, g4, g5, g6, g7, X32, c2, X33, c3, X34. Since the values ​​of interpolation points a2, b2, and c2 in the second set of matrix elements will not exceed the larger of X12 and X13 in the input feature map, respectively; the values ​​of interpolation points a3 and b3 will not exceed the larger of X13 and X14 in the input feature map; the values ​​of interpolation points b2, b3, and c4 will not exceed the larger of X23 and X24 in the input feature map; the values ​​of interpolation points c2, c3, and f3 will not exceed the larger of X33 and X34 in the input feature map; the values ​​of interpolation points f3, f5, and f7 will not exceed the larger of X12 and X22 in the input feature map; and the values ​​of interpolation points f5, f6, f7, and f8 will not exceed the larger of X13 and X23 in the input feature map. The larger value in X14 and X24 of the feature map, the value of interpolation point f4 will not exceed the larger value in interpolation points a2 and b2, and the value of interpolation point f6 will not exceed the larger value in interpolation points a3 and b3; the value of interpolation point g3 will not exceed the larger value in X22 and X32 of the input feature map, the value of interpolation point g5 will not exceed the larger value in X23 and X33 of the input feature map, the value of interpolation point g7 will not exceed the larger value in X24 and X34 of the input feature map, the value of interpolation point g4 will not exceed the larger value in interpolation points b2 and c2, and the value of interpolation point g6 will not exceed the larger value in interpolation points b3 and c3. Therefore, the result of finding the maximum value of the elements of the second group of matrices is the same as the result of finding the maximum value of X12, X13, X14, X22, X23, X24, X32, X33, and X34.

[0122] Similarly, the BPU moves sequentially from left to right and top to bottom with the second pooling stride to determine each matrix element in the magnified feature map for pooling operations, and performs max pooling on each matrix element to obtain the pooled output feature map, which has a size of 3*4. In other words, the output feature map obtained by using a 5*5 magnified pooling kernel and a second pooling stride supported by the neural network accelerator to perform max pooling on the magnified feature map is the same as the output feature map obtained by using a 3*3 pooling kernel and a pooling layer stride to perform max pooling on the input feature map.

[0123] The method provided in this application for performing pooling operations on pooling layers in a neural network model using a neural network accelerator, wherein when the pooling operation type of the pooling layer is max pooling, if the second pooling step size supported by the neural network accelerator is greater than the pooling step size of the pooling layer, then the input feature map can be enlarged, and pooling operations can be performed on the enlarged feature map based on the second pooling step size supported by the neural network accelerator. This application allows the neural network accelerator to perform pooling operations with the first pooling step size using its supported second pooling step size without changing the hardware parameters of the neural network accelerator.

[0124] Figure 14 An instruction generation apparatus provided in the embodiments of this application, such as Figure 14 As shown, the instruction generation device 1400 includes a first determination module 1401, a second determination module 1402, and an instruction generation module 1403.

[0125] The first determining module 1401 is used to determine the hardware parameters of the neural network accelerator that performs pooling operations.

[0126] The second determining module 1402 is used to determine the pooling parameters of the pooling layer for which pooling operations are performed. The pooling parameters include the first pooling stride, the size of the input feature map of the pooling layer, the size of the pooling kernel, and the type of pooling operation.

[0127] The instruction generation module 1403 is used to generate executable instructions for the neural network accelerator based on the pooling parameters and hardware parameters.

[0128] In some embodiments, the instruction generation module 1403 is specifically used to determine the support relationship between the first pooling step size and hardware parameters based on the type of pooling operation; and to generate instructions executable by the neural network accelerator based on the support relationship.

[0129] For example, when the pooling operation is average pooling, the supporting relationships include that the hardware parameters of the neural network accelerator do not support pooling operations with a first pooling stride, and that the hardware parameters of the neural network accelerator support convolution operations with the same stride as the first pooling stride. The instruction generation module 1403 is specifically used to determine the convolution kernel and a first value based on this supporting relationship and the size of the pooling kernel. The size of the convolution kernel is the same as the size of the pooling kernel. Based on the convolution kernel and the first value, it generates convolution operation instructions and an average value calculation instruction executable by the neural network accelerator. The convolution operation instruction includes a convolution stride, which is the first pooling stride. The convolution operation instruction is used to perform a convolution operation between the matrix elements participating in the pooling operation in the input feature map and the convolution kernel to obtain the convolution output value of the matrix elements. The average value calculation instruction is used to divide the convolution output value of the matrix elements by the first value to obtain the average pooling output value of the matrix elements.

[0130] For example, when the pooling operation is max pooling, the support relationship includes that the hardware parameters of the neural network accelerator do not support pooling operations with a first pooling step size. The instruction generation module 1403 is specifically used to determine the enlarged pooling kernel size based on the support relationship and the size of the pooling kernel; and to generate a feature map enlargement instruction and a pooling operation instruction executable by the neural network accelerator based on the enlarged pooling kernel size. The feature map enlargement instruction is used to enlarge the input feature map using a bilinear interpolation algorithm according to the size of the first and second pooling step sizes, obtaining an enlarged feature map, where the second pooling step size is the pooling step size supported by the hardware parameters. The pooling operation instruction includes a second pooling step size and is used to perform pooling operations on the matrix elements participating in the pooling operation in the enlarged feature map based on the enlarged pooling kernel size to obtain the max pooling output value of the matrix elements.

[0131] Figure 15 A pooling operation device provided in the embodiments of this application, such as Figure 15 As shown, the pooling computing device 1500 includes a determination module 1501 and a processing module 1502.

[0132] The determination module 1501 is used to determine the pooling parameters of the pooling layers in the neural network model that the neural network accelerator needs to execute. These pooling parameters include the first pooling stride, the type of pooling operation of the pooling layer, and the size of the pooling kernel corresponding to the pooling layer.

[0133] The processing module 1502 is used to process the input feature map of the pooling layer based on the pooling parameters to obtain the pooled output feature map.

[0134] In some embodiments, the pooling operation is of the average pooling type. The processing module 1502 is specifically used to determine the convolution stride supported by the neural network accelerator, which is the same as the first pooling stride; based on the convolution stride and the pooling kernel size, determine the convolution kernel and the first value, wherein the size of the convolution kernel is the same as the size of the pooling kernel, and the values ​​of all parameters in the convolution kernel are the same; by performing a convolution operation on the input feature maps of the convolution kernel and the pooling layer, a convolutional feature map is obtained; and by dividing the parameters in the convolutional feature map by the first value, a pooled output feature map is obtained.

[0135] In some embodiments, the pooling operation is max pooling. The processing module 1502 is specifically used to determine the second pooling step size supported by the neural network accelerator, wherein the second pooling step size is greater than the first pooling step size; determine the magnification factor based on the first pooling step size and the second pooling step size; magnify the input feature map using a bilinear interpolation algorithm based on the magnification factor to obtain a magnified feature map; determine the magnified pooling kernel size based on the magnification factor and the pooling kernel size; and perform max pooling operation on the magnified feature map based on the second pooling step size and the magnified pooling kernel size to obtain a pooled output feature map.

[0136] Figure 16 A structural diagram of an electronic device provided in an embodiment of this application, such as... Figure 16 As shown, the electronic device 1600 includes one or more processors 1601 and memory 1602.

[0137] The processor 1601 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1600 to perform desired functions.

[0138] The memory 1602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1601 may execute the program instructions to implement the instruction generation method or pooling operation method of the various embodiments of this application described above, and / or other desired functions.

[0139] In one example, the electronic device 1600 may also include an input device 1603 and an output device 1604, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0140] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device 1600 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 1600 may include any other suitable components depending on the specific application.

[0141] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the instruction generation method or pooling operation method according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0142] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0143] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the instruction generation method or pooling operation method according to various embodiments of this application described in the "Exemplary Methods" section above.

[0144] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0146] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0147] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0148] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0149] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for generating instructions for pooling operations in a neural network accelerator, comprising: Determine the hardware parameters of the neural network accelerator that performs pooling operations; Determine the pooling parameters of the pooling layer to perform the pooling operation; The pooling parameters include the first pooling step size, the size of the input feature map of the pooling layer, the size of the pooling kernel, and the type of pooling operation; Based on the type of pooling operation, determine the support relationship between the first pooling step size and the hardware parameters; Based on the aforementioned support relationship, executable instructions for the neural network accelerator are generated; Wherein, if the pooling operation is max pooling and the support relationship includes situations where the hardware parameters do not support pooling operations with the first pooling step size, the step of generating executable instructions for the neural network accelerator based on the support relationship includes: Based on the support relationship and the size of the pooling kernel, the enlarged pooling kernel size is determined; Based on the enlarged pooling kernel size, feature map amplification instructions and pooling operation instructions executable by the neural network accelerator are generated.

2. The method according to claim 1, wherein, The pooling operation is of the average pooling type, and the support relationship includes that the hardware parameters do not support pooling operations with the first pooling step size, and the hardware parameters support convolution operations with the same length as the first pooling step size.

3. The method according to claim 2, wherein, The step of generating executable instructions for the neural network accelerator based on the support relationship includes: Based on the support relationship and the size of the pooling kernel, the convolution kernel and the first value are determined; the size of the convolution kernel is the same as the size of the pooling kernel. Based on the convolution kernel and the first value, convolution operation instructions and average calculation instructions executable by the neural network accelerator are generated; the convolution operation instructions include a convolution stride, which is the first pooling stride, and the convolution operation instructions are used to perform convolution operations on the matrix elements participating in the pooling operation in the input feature map with the convolution kernel to obtain the convolution output value of the matrix elements; the average calculation instructions are used to divide the convolution output value of the matrix elements by the first value to obtain the average pooling output value of the matrix elements.

4. The method according to claim 1, wherein, The feature map magnification instruction is used to magnify the input feature map using a bilinear interpolation algorithm based on the size of the first pooling step and the second pooling step, to obtain a magnified feature map. The second pooling step is the pooling step supported by the hardware parameters. The pooling operation instruction includes the second pooling step and is used to perform pooling operations on the matrix elements participating in the pooling operation in the magnified feature map based on the magnified pooling kernel size, so as to obtain the maximum pooling output value of the matrix elements.

5. A method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator, comprising: The pooling parameters of the pooling layer in the neural network model that the neural network accelerator needs to execute are determined. The pooling parameters include the first pooling step size, the type of pooling operation of the pooling layer, and the pooling kernel size corresponding to the pooling layer. Based on the pooling parameters, the input feature map of the pooling layer is processed to obtain the pooled output feature map; Wherein, when the pooling operation is max pooling, the process of processing the input feature map of the pooling layer based on the pooling parameters to obtain the pooled output feature map includes: Determine a second pooling step size supported by the neural network accelerator, wherein the second pooling step size is greater than the first pooling step size; Based on the first pooling step size and the second pooling step size, the input feature map of the pooling layer is magnified to obtain the magnified feature map; The amplified feature map is pooled based on the second pooling step size to obtain the pooled output feature map.

6. The method according to claim 5, wherein, The pooling operation is of the average pooling type. The process of processing the input feature map of the pooling layer based on the pooling parameters to obtain the pooled output feature map includes: Determine the convolution stride supported by the neural network accelerator, wherein the convolution stride is the same as the first pooling stride; Based on the convolution stride and the pooling kernel size, the convolution kernel and the first value are determined, wherein the size of the convolution kernel is the same as the size of the pooling kernel; By performing convolution operations on the input feature maps of the convolution kernel and the pooling layer, a convolutional feature map is obtained; Divide the parameters in the convolutional feature map by the first value to obtain the pooled output feature map.

7. The method according to claim 5, wherein, The step of magnifying the input feature map of the pooling layer based on the first pooling step size and the second pooling step size to obtain a magnified feature map includes: The amplification factor is determined based on the first pooling step size and the second pooling step size; Based on the magnification factor, the input feature map is magnified using a bilinear interpolation algorithm to obtain the magnified feature map; Correspondingly, the step of performing pooling operations on the amplified feature map based on the second pooling step size to obtain the pooled output feature map includes: Based on the magnification factor and the pooling kernel size, the magnified pooling kernel size is determined; Based on the second pooling step size and the enlarged pooling kernel size, a pooling operation is performed on the enlarged feature map to obtain the pooled output feature map.

8. An instruction generation apparatus for pooling operations in a neural network accelerator, comprising: The first determining module is used to determine the hardware parameters of the neural network accelerator that performs pooling operations; The second determining module is used to determine the pooling parameters of the pooling layer for pooling operations; the pooling parameters include the first pooling step size, the size of the input feature map of the pooling layer, the size of the pooling kernel, and the type of pooling operation; An instruction generation module is configured to determine a support relationship between the first pooling step size and the hardware parameters based on the type of the pooling operation; and to generate executable instructions for the neural network accelerator based on the support relationship. Where the type of the pooling operation is max pooling, and the support relationship includes cases where the hardware parameters do not support pooling operations with the first pooling step size, generating executable instructions for the neural network accelerator based on the support relationship includes: determining an amplified pooling kernel size based on the support relationship and the size of the pooling kernel; and generating feature map amplification instructions and pooling operation instructions executable by the neural network accelerator based on the amplified pooling kernel size.

9. A computer-readable storage medium storing a computer program for executing the instruction generation method for pooling operations in a neural network accelerator as described in any one of claims 1-4, or for executing the method for performing pooling operations on pooling layers in a neural network model using a neural network accelerator as described in any one of claims 5-7.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the instruction generation method for pooling operations in a neural network accelerator as described in any one of claims 1-4, or to implement the method for pooling operations in a pooling layer of a neural network model using a neural network accelerator as described in any one of claims 5-7.

Images