A method and device for generating a neural network instruction sequence

Abstract

The present invention discloses a method and device for generating an instruction sequence of a neural network. First, according to the feature map dimension of the global average pooling layer in the neural network, the feature map dimension of the depthwise separable convolution in the neural network is determined. Then, according to the input feature map dimension of the global average pooling layer, the dimension of the convolution kernel used for the depthwise separable convolution is determined. Then, according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, an instruction sequence corresponding to the execution of the neural network is generated. It can be seen that the present invention determines the feature map dimension and the dimension of the convolution kernel of the depthwise separable convolution according to the feature map dimension of the global average pooling layer, and generates a corresponding instruction sequence. Therefore, when encountering the operation of the global average pooling layer, the above instruction sequence can be executed to use the depthwise separable convolution to replace the operation of the global average pooling layer, thereby avoiding changing the chip hardware structure.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for generating an instruction sequence of a neural network. Background Art

[0002] Usually, pooling operations are used in most neural networks. However, for chips, the more pooling operations they support, the greater the cost of chip area and power consumption. If pooling operations are added to a chip structure that does not support pooling operations, additional hardware logic will be added to the chip, thereby changing the hardware structure of the chip.

[0003] Therefore, how to realize pooling operation without changing the chip hardware structure is a problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the above technical problems, this application is proposed.

[0005] According to one aspect of the present application, a method for generating an instruction sequence of a neural network is provided, the method comprising: determining the feature map dimension of a depthwise separable convolution in the neural network according to the feature map dimension of a global average pooling layer in the neural network; determining the dimension of a convolution kernel for the depthwise separable convolution according to the input feature map dimension of the global average pooling layer; and generating an instruction sequence corresponding to executing the neural network according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, the instruction sequence including operation instructions of the global average pooling layer of the neural network.

[0006] According to another aspect of the present application, a method for operating a neural network is provided, wherein the neural network includes an instruction sequence generated according to the above-mentioned claim; the method includes: obtaining an input feature map after being processed by a convolutional layer in the neural network; wherein the input feature map includes at least the number of channels; determining the number of convolution kernels according to the number of channels of the input feature map; wherein one channel corresponds to one convolution kernel; in each channel of the input feature map, convolving each channel according to its own convolution kernel to obtain an output value of each channel; wherein the output value of each channel is the same as the output value of the channel obtained by a global average pooling operation.

[0007] According to another aspect of the present application, a device for generating an instruction sequence of a neural network is provided, comprising: a first determination module, used to determine the feature map dimension of a depthwise separable convolution in the neural network according to the feature map dimension of a global average pooling layer in the neural network; a second determination module, used to determine the dimension of a convolution kernel used for the depthwise separable convolution according to the input feature map dimension of the global average pooling layer; a generation module, used to generate an instruction sequence corresponding to executing the neural network according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, wherein the instruction sequence includes operation instructions of the global average pooling layer of the neural network.

[0008] According to another aspect of the present application, there is provided a device for operating a neural network, wherein the neural network includes an instruction sequence generated according to the above-mentioned claim; the device includes: an obtaining unit, used to obtain an input feature map after being processed by a convolution layer in the neural network; wherein the input feature map includes at least the number of channels; a determining unit, used to determine the number of convolution kernels according to the number of channels of the input feature map; wherein one channel corresponds to one convolution kernel; a convolution unit, used to convolve each channel of the input feature map according to the convolution kernel of each channel to obtain an output value of each channel; wherein the output value of each channel is the same as the output value of the channel obtained after a global average pooling operation.

[0009] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the method described above.

[0010] According to another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor is caused to execute the method described above.

[0011] Compared with the prior art, the present application first determines the feature map dimension of the depthwise separable convolution in the neural network according to the feature map dimension of the global average pooling layer in the neural network. Then, according to the input feature map dimension of the global average pooling layer, the dimension of the convolution kernel used for the depthwise separable convolution is determined. Then, according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, an instruction sequence corresponding to the execution of the neural network is generated. The hardware instruction sequence includes the operation instructions of the global average pooling layer of the neural network, that is, in the present invention, the feature map dimension and the dimension of the convolution kernel of the depthwise separable convolution are determined according to the feature map dimension of the global average pooling layer, and a corresponding instruction sequence is generated. Therefore, when encountering the operation of the global average pooling layer, the above instruction sequence can be executed to use the depthwise separable convolution to replace the operation of the global average pooling layer, thereby avoiding changing the chip hardware structure.

[0012] The present application obtains the input feature map after being processed by the convolution layer in the neural network, and then determines the number of convolution kernels according to the number of channels of the input feature map, and in each channel of the input feature map, convolves each channel according to the convolution kernel of each channel to obtain the output value of each channel. It can be seen that the present application uses depthwise separable convolution to process the input feature map that is originally to enter the pooling operation after being processed by the convolution layer, and can use depthwise separable convolution to replace the operation of the global average pooling layer, thereby avoiding changing the chip hardware structure.

[0013] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 It is a flowchart of a method for generating an instruction sequence of a neural network provided by an exemplary embodiment of the present application.

[0016] Figure 2 It is a schematic diagram of an input feature map of a global average pooling layer provided by another exemplary embodiment of the present application.

[0017] Figure 3It is a schematic diagram of a convolution kernel provided by an exemplary embodiment of the present application.

[0018] Figure 4A It is a flowchart of determining the dimension of a convolution kernel for depthwise separable convolution provided by an exemplary embodiment of the present application.

[0019] Figure 4B It is a schematic diagram of a depth-wise separable convolution operation provided by an exemplary embodiment of the present application.

[0020] Figure 5 It is a flowchart of a method for operating a neural network provided by an exemplary embodiment of the present application.

[0021] Figure 6 It is an example block diagram of an instruction sequence generating device for a neural network provided by an exemplary embodiment of the present application.

[0022] Figure 7 It is an example block diagram of the second determination module 602 provided by an exemplary embodiment of the present application.

[0023] Figure 8 It is an example block diagram of an apparatus for neural network operation provided by an exemplary embodiment of the present application.

[0024] Fig. 9 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0025] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0026] Application Overview

[0027] As mentioned above, the chip supports limited operations in terms of hardware structure. If additional operations are added, the hardware structure of the chip itself will change. Taking depthwise separable convolution as an example, if the chip supports global average pooling operations in addition to depthwise separable convolution operations, the hardware logic of the chip will inevitably increase and the hardware structure of the chip will change.

[0028] Taking the above problems into consideration, the present application has developed a method for generating an instruction sequence for a neural network, which aims to use depth-wise separable convolution to replace the global average pooling operation without changing the chip hardware structure, based on the fact that the chip originally supports depth-wise separable convolution but does not support global average pooling operation. Since the result of the depth-wise separable convolution operation is the same as the result of the global average pooling operation, the depth-wise separable convolution can be used to replace the calculation during the global average pooling operation, thereby enabling the chip to support global average pooling calculations of any size without changing the chip hardware structure.

[0029] It is worth noting that the global average pooling calculation supporting any size in this embodiment means that for input feature maps of any height, width and number of channels, depthwise separable convolution can be used to calculate them instead of the global average pooling operation.

[0030] Exemplary Methods

[0031] Figure 1 This is a flowchart of a method for generating an instruction sequence for a neural network provided by an exemplary embodiment of the present application. This embodiment can be applied to an AI chip, and can enable the AI ​​chip to support global average pooling operations and complete classification tasks without increasing additional hardware costs. It can then be applied to many fields such as autonomous driving, smart retail, and industrial robots. The AI ​​chip in this embodiment refers to a SOC chip that includes a neural network computing accelerator.

[0032] The neural network instruction sequence generation method described in one or more embodiments of the present application is as follows Figure 1 As shown, the following steps are included:

[0033] Step 101, determining the feature map dimension of the depthwise separable convolution in the neural network according to the feature map dimension of the global average pooling layer in the neural network.

[0034] Among them, the neural network of this embodiment includes: deep neural network (DDN), convolutional neural network (CNN), recurrent neural network (RNN), genetic adversarial neural network (GAN), etc.

[0035] In neural networks, the main function of the global average pooling layer is to compress a feature map with several channels into a feature vector with only one pixel. Taking convolutional neural networks as an example, using the global average pooling layer in convolutional neural networks can effectively reduce the number of parameters, making the model more robust and more resistant to overfitting. Currently, many neural networks used for classification use the global average pooling layer.

[0036] The feature maps of the global average pooling layer include: input feature map and output feature map. The feature map dimensions include: feature map height, feature map width, and feature map channel number. Taking the input feature map as an example, the input feature map dimensions include: input feature map height, input feature map width, and input feature map channel number. Taking the output feature map as an example, the output feature map dimensions include: output feature map height, output feature map width, and output feature map channel number.

[0037] For example, see Figure 2 , the height, width, and number of channels of the input feature map of the global average pooling layer are: H in , W in , C in Each layer in the input feature map represents a channel. Global average pooling is to calculate the arithmetic mean of each layer and turn all the pixels of a channel into a single point. In each channel, a single pixel constitutes a feature map with a height and width of 1. Therefore, the height, width, and number of channels of the output feature map of global average pooling are 1, 1, and C respectively. in .

[0038] The above is an introduction to the global average pooling layer and its feature map dimensions. For depthwise separable convolution, it also includes input feature maps and output feature maps. Similarly, the input feature map dimensions of depthwise separable convolution include: input feature map height, input feature map width, and input feature map channel number. The output feature map dimensions of depthwise separable convolution include: output feature map height, output feature map width, and output feature map channel number.

[0039] As for how to determine the feature map dimension of the depthwise separable convolution, there will be a detailed introduction later, so I will not go into details here.

[0040] Step 102: Determine the dimension of the convolution kernel used for depthwise separable convolution according to the dimension of the input feature map of the global average pooling layer.

[0041] Specifically, the input feature map dimension of the global average pooling layer has been introduced in the above embodiment. For the convolution kernel of this embodiment, its function is to perform convolution calculation with the input feature map of the depthwise separable convolution, so that the result of the convolution calculation is the same as the result of the global average pooling operation. Therefore, during the global average pooling operation, the depthwise separable convolution can be used to replace the calculation to avoid changing the chip hardware structure, and the chip can support global average pooling calculation of any size without changing the chip hardware structure.

[0042] For the convolution kernel used for depthwise separable convolution, the dimensions of the convolution kernel include: the height of the convolution kernel, the width of the convolution kernel, the number of channels of the convolution kernel, the value of the convolution kernel, and the number of convolution kernels. Among them, the number of channels of the convolution kernel is 1, and the value of the convolution kernel refers to the value of the tensor data of the convolution kernel. Since the convolution kernel can be regarded as a multidimensional tensor composed of a specific shape, the value of the convolution kernel refers to each tensor data in this multidimensional tensor. If the convolution kernel is regarded as a graph with height and width, the value of the convolution kernel refers to the value of each pixel in the graph. See Figure 3 , the height, width, number of channels, and values ​​of the convolution kernel are: H in , W in 1. kernel[C k ][h][w].

[0043] Step 103: Generate an instruction sequence corresponding to the neural network for executing the instruction sequence according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel.

[0044] The instruction sequence is obtained according to the dimension of the feature map of the depthwise separable convolution and the dimension of the convolution kernel, as the operation instruction of the global average pooling layer of the neural network. Therefore, when performing the global average pooling layer operation, executing the instruction sequence can convert the global average pooling layer operation into a depthwise separable convolution operation.

[0045] In the above Figure 1 On the basis of the illustrated embodiment, as an optional implementation of this embodiment, since the global average pooling is to be replaced by the depthwise separable convolution calculation, it is necessary to ensure that the input and output feature map dimensions of the depthwise separable convolution are the same as the input and output feature map dimensions of the global average pooling. Therefore, in the implementation process of step 101, in which the feature map dimension of the depthwise separable convolution in the neural network is determined according to the feature map dimension of the pooling layer in the neural network, the input feature map dimension of the global average pooling layer is determined as the input feature map dimension of the depthwise separable convolution. The output feature map dimension of the global average pooling layer is determined as the output feature map dimension of the depthwise separable convolution.

[0046] Specifically, the height, width, and number of channels of the input feature map of the global average pooling layer are determined as the height, width, and number of channels of the input feature map of the depthwise separable convolution. Similarly, the height, width, and number of channels of the output feature map of the global average pooling layer are determined as the height, width, and number of channels of the output feature map of the depthwise separable convolution. For example, assuming that the height, width, and number of channels of the input feature map of the global average pooling layer are H in , W in , C inThe height, width, and number of channels of the output feature map of the global average pooling layer are 1, 1, and C respectively. in . Then replace the global average pooling layer with a depthwise separable convolution. The height, width, and number of channels of the input and output feature maps of the depthwise separable convolution are consistent with those of the global average pooling layer. Then the height, width, and number of channels of the input feature map of the depthwise separable convolution are H in , W in , C in The height, width, and number of channels of the output feature map of the depthwise separable convolution are 1, 1, and C respectively. in .

[0047] In the above Figure 1 Based on the embodiment shown, as an optional implementation of this embodiment, since the dimensions of the convolution kernel include: the height of the convolution kernel, the width of the convolution kernel, the number of channels of the convolution kernel, the value of the convolution kernel and the number of convolution kernels. Therefore, in the implementation process of determining the dimensions of the convolution kernel used for depthwise separable convolution according to the input feature map dimensions of the global average pooling layer in step 102, refer to Figure 4A There is no restriction on the order of the steps listed in the method.

[0048] Step 401, determining the height and width of the convolution kernel based on the input feature map height and input feature map width of the global average pooling layer.

[0049] Specifically, the input feature map height and input feature map width of the global average pooling layer are respectively determined as the height and width of the convolution kernel, so that the height and width of the convolution kernel are respectively consistent with the input feature map height and input feature map width of the global average pooling layer.

[0050] Step 402, determining the number of the convolution kernels according to the number of channels of the input feature map of the global average pooling layer, so that the number of the convolution kernels is the same as the number of channels of the input feature map.

[0051] Since the convolution kernel is used for depth-wise separable convolution calculation, its number of channels is 1. The number of convolution kernels is determined by the number of input feature map channels of the global average pooling layer.

[0052] Step 403, determining the value of the convolution kernel according to the input feature map height and input feature map width of the global average pooling layer.

[0053] Further, the product of the height and width of the input feature map of the global average pooling layer is determined. The value of the convolution kernel is determined based on the inverse of the product. For example, assuming that the height and width of the input feature map of the global average pooling layer are H in , W in. Then the value of the convolution kernel kernel[C k ][h][w]=1 / (H in *W in ). It is worth noting that the values ​​of all convolution kernels are equal. The value of the convolution kernel determined in this way can be convolved with the corresponding input feature map in the operation of depthwise separable convolution, and can make the output result after convolution calculation consistent with the output result of global average pooling.

[0054] As an optional embodiment, if the height and width of the convolution kernel are determined, the value of the convolution kernel can also be determined using the height and width of the convolution kernel. Since the height and width of the convolution kernel are respectively consistent with the height and width of the input feature map of the global average pooling layer, the obtained value is consistent with the value determined in the above embodiment, which is 1 / (H in *W in ).

[0055] The above is an embodiment of determining the value of the convolution kernel based on the input feature map height and input feature map width of the global average pooling layer. Since the value of the convolution kernel is related to the precision supported by the target hardware, after obtaining the value of the convolution kernel, the value of the convolution kernel can also be adjusted according to the precision supported by the target hardware. Specifically, it can be determined by the following steps: First, determine the precision supported by the target hardware (that is, the target chip, which refers to the chip running the method). Secondly, adjust the value of the convolution kernel according to the precision supported by the target hardware. In the adjustment process, first determine the ratio of the value and precision of the convolution kernel; then round the ratio, and then multiply the value obtained after rounding by the precision to obtain the product, and use the product as the value of the adjusted convolution kernel. If the precision is J, the value of the adjusted convolution kernel is k'. Then according to the formula Get the adjusted convolution kernel value. For example, many deep learning acceleration chips on the market only support 8-bit reasoning, and the 8-bit representation accuracy is The convolution kernel value is equal to Then the value of the adjusted convolution kernel is This method is suitable for scenarios where the neural network model is a fixed-point model or the target hardware supports fixed-point inference.

[0056] In the above Figure 1On the basis of the shown embodiment, as an optional implementation manner of this embodiment, since the number of channels of the input feature map is equal to the number of convolution kernels, and each channel corresponds to its own convolution kernel, in step 103, according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, an instruction sequence corresponding to the neural network is generated. According to the input feature map of the depthwise separable convolution and its feature map dimension, the convolution kernel and its dimension, the convolution operation process, and the output feature map of the depthwise separable convolution and its feature map dimension, an instruction sequence is generated.

[0057] During the convolution operation, the input feature map of the depth-separable convolution is convolved according to the input feature map of the depth-separable convolution and its feature map dimension, the convolution kernel and its dimension, to obtain the output feature map of the depth-separable convolution. Specifically, in each channel of the input feature map of the depth-separable convolution, the following processing is performed: the channel and its convolution kernel are convolved to obtain the output value of the channel. Among them, the output value is the same as the output value obtained by the global average pooling operation of the channel. That is, after the convolution calculation, all the pixels of the channel become a point of the result, and the output value of the channel is equal to the average value of all the pixels of the channel. It can be seen that after each channel of the input feature map is convolved, it is transformed into an output feature map with a height and width of 1, and the pixel value of the output feature map in each channel is the output value obtained by each channel through the above-mentioned convolution operation, and is the same as the output value obtained by each channel through the global average pooling operation. Further, the output feature map is input into the fully connected layer for processing.

[0058] Specifically, for the convolution processing of each channel, in the height direction, the product of each row in the height direction is determined according to the step length in the height direction when each row and the corresponding convolution kernel move in the channel; the product of each row in the height direction and the coordinate value of each row in the current height direction are combined to obtain the first input parameter of each row in the height direction; in the width direction, the product of each column in the width direction is determined according to the step length in the width direction when each column and the corresponding convolution kernel move in the channel; the product of each column in the width direction and the coordinate value of each row in the current width direction are combined to obtain the second input parameter of each row in the width direction; the output value of the channel is obtained by weighted summation of the first input parameter, the second input parameter and the corresponding convolution kernel. If expressed in formula, then through the formula Perform convolution operation. Among them, H i Indicates the height direction H i Row (0 <= H i <=H), W j Indicates the W jColumn (0 <= W j <W),C k Indicates C k Channels (0 <= C k <C),output[H i ][W j ][C k ] indicates the C k h, w represent the coordinate values ​​of the height and width directions when summing, S h Indicates the step length in the height direction when the convolution kernel moves, S w Indicates the width step size when the convolution kernel moves. input[Hi*S h +h][W j *S w +w][C k ] indicates the H i *S h +h line W j *S w +w column C k The value of each channel. kernel[C k ][h][w] indicates the C k The value of the hth row and wth column of the convolution kernel, kernel[C k ][h][w]=1 / (H in *W in ), is a constant. It represents the weighted summation of the local input feature map according to the convolution kernel.

[0059] Please see below Figure 4B , the length, width, and number of channels of the input feature map of the depthwise separable convolution are: H in =W in =12, C in =3. The length, width, number of channels, and value of the convolution kernel are: in =W in =12, number of channels = 1, kernel[C k ][h][w]=1 / (H in *W in )=0.028. Use 3 convolution kernels with a channel number of 1 to convolve with the data of each channel in the input feature map to obtain the output value of each channel, and the output value of each channel is the same as the output value obtained by the global average pooling operation of the channel. Therefore, after each channel of the input feature map is convolved, it is converted into an output feature map with a height and width of 1, and the number of channels is the same as the number of channels of the input feature map.

[0060] The above is the specific implementation process of using depthwise separable convolution to implement global average pooling operation and generate an instruction sequence. Based on this instruction sequence, a parameter file for executing the neural network is generated. It is worth noting that this instruction sequence is only a partial instruction sequence of the neural network during the compilation process, as an intermediate representation of the compilation process. On this basis, combined with other instruction sequences compiled before and after this instruction sequence, the hardware instruction sequence required to execute the neural network, i.e. the parameter file, is generated.

[0061] It is worth noting that the structure of the neural network in the embodiment of the present invention has not changed, and still includes: convolution layer, pooling layer, and fully connected layer. It is just that when operating in the pooling layer, the processing logic of the convolution layer is reused, and the global average pooling operation is implemented using depthwise separable convolution. The neural network itself has not changed at all. The replaced depthwise separable convolution is mathematically equivalent to global average pooling, and there is no difference in the model inference results.

[0062] Because depth-wise separable convolution is used to replace the global average pooling operation, this method can be applied to chips that support depth-wise separable convolution but do not support global average pooling operations, and can enable the chip to support global average pooling calculations of any size without adding additional hardware.

[0063] Taking practical applications as an example, for example, the Resnet model commonly used in classification tasks, in order to improve the operation speed, the global average pooling operation is often used in the last layer in practical applications. However, for some chips that do not support global average pooling, the Resnet model cannot run on them. Using the technical solution in the embodiment of the present application, the convolution kernel can be constructed during the compilation process of the Resnet model, and the global average pooling can be replaced by the depthwise separable convolution without changing the operation results of the model, that is, the operation results of the depthwise separable convolution and the global average pooling are the same.

[0064] Based on the same inventive concept and beneficial effects as one or more of the above embodiments, the following embodiment discloses a method for operating a neural network, wherein the neural network includes an instruction sequence generated according to any of the above embodiments.

[0065] Figure 5 This is a flowchart of a method for implementing a neural network operation on an AI chip provided by an exemplary embodiment of the present application. This method can be applied to many fields such as autonomous driving, smart retail, and industrial robots.

[0066] The method of operating the neural network described in one or more embodiments of the present application is as follows Figure 5 As shown, the following steps are included:

[0067] Step 501, obtaining an input feature map after being processed by the convolutional layer in the neural network.

[0068] Among them, the neural network includes: a convolution layer, a pooling layer, and a fully connected layer. After the convolution layer in the neural network is processed, the input feature map obtained can be regarded as the input feature map of the global average pooling layer, and a pooling operation is performed on it. In an embodiment of the present invention, a convolution kernel is constructed to perform convolution processing on the input feature map, and the processing result is the same as the processing result of the global average pooling, so as to replace the global average pooling operation. This will be described in detail later and will not be repeated here.

[0069] The dimensions of the input feature map include at least: width, height, and number of channels.

[0070] Step 502: Determine the number of convolution kernels according to the number of channels of the input feature map.

[0071] Among them, since the input feature map needs to be subjected to depthwise separable convolution in the pooling layer, the number of convolution kernels depends on the number of channels of the input feature map, and one channel corresponds to one convolution kernel.

[0072] For the convolution kernel, see Figure 3 The dimensions of the convolution kernel include: the height of the convolution kernel, the width of the convolution kernel, the number of channels of the convolution kernel, the value of the convolution kernel and the number of convolution kernels. Among them, the number of channels of the convolution kernel is 1, and the value of the convolution kernel refers to the value of the tensor data of the convolution kernel. Since the convolution kernel can be regarded as a multidimensional tensor composed of a specific shape, the parameters here refer to each tensor data in this multidimensional tensor.

[0073] Since the dimensions of the convolution kernel include: the height of the convolution kernel, the width of the convolution kernel, the number of channels of the convolution kernel, the value of the convolution kernel and the number of convolution kernels. Therefore, it is necessary to determine the height of the convolution kernel, the width of the convolution kernel and the value of the convolution kernel. Specifically, the height and width of the convolution kernel are determined according to the height and width of the input feature map, so that the height and width of the convolution kernel are consistent with the height and width of the input feature map, respectively. The value of the convolution kernel is determined according to the height and width of the input feature map. Further, the value of the convolution kernel corresponding to each channel is the same, wherein the value is the inverse of a product, and the product is obtained by multiplying the height and width of the convolution kernel; the height and width of the convolution kernel are the same as the height and width of the input feature map. The specific implementation method of determining the value of the convolution kernel has been described in detail in the aforementioned embodiment, so it will not be repeated here. At present, the value of the convolution kernel can be adjusted according to the accuracy of the target hardware, and the specific adjustment method has also been described in detail in the aforementioned embodiment, so it will not be repeated here. It is worth noting that, generally speaking, there is no order of precedence for the steps to determine the height, width, value, and number of convolution kernels. However, if the method for determining the value of the convolution kernel is related to the height and width of the convolution kernel, it is necessary to first determine the height and width of the convolution kernel, and then determine the value of the convolution kernel based on this.

[0074] Step 503: In each channel of the input feature map, convolution is performed on each channel according to the convolution kernel of each channel to obtain an output value of each channel.

[0075] Among them, the output value of each channel is the same as the output value obtained by the global average pooling operation of the channel. That is, after the convolution calculation, all the pixels of the channel become one point of the result, and the output value of the channel is equal to the average value of all the pixels of the channel. It can be seen that after each channel of the input feature map is convolved, it is transformed into an output feature map with a height and width of 1, and the pixel value of each channel of the output feature map is the output value obtained by the above convolution operation. Further, the output feature map will be input into the softmax layer for processing.

[0076] For the convolution processing of each channel, in the height direction, the product of each row in the height direction is determined according to the step length in the height direction when each row and the corresponding convolution kernel move; the product of each row in the height direction and the coordinate value of each row in the current height direction are combined to obtain the first input parameter of each row in the height direction; in the width direction, the product of each column in the width direction is determined according to the step length in the width direction when each column and the corresponding convolution kernel move; the product of each column in the width direction and the coordinate value of each row in the current width direction are combined to obtain the second input parameter of each row in the width direction; the output value of the channel is obtained by weighted summation of the first input parameter, the second input parameter and the corresponding convolution kernel. If expressed in formula, then through the formula Perform convolution operation. Among them, H i Indicates the height direction H i Row (0 <= H i <=H), W j Indicates the W j Column (0 <= W j <W),C k Indicates C k Channels (0 <= C k <C),output[H i ][W j ][C k ] indicates the C k h, w represent the coordinate values ​​of the height and width directions when summing, S h Indicates the step length in the height direction when the convolution kernel moves, S w Indicates the width step size when the convolution kernel moves. input[Hi*S h +h][W j *S w +w][C k ] indicates the H i *S h +h line W j *S w +w column C k The value of each channel. kernel[C k ][h][w] indicates the C k The value of the hth row and wth column of the convolution kernel, kernel[C k ][h][w]=1 / (H in *W in ), is a constant. It represents the weighted summation of the local input feature map according to the convolution kernel.

[0077] As an optional embodiment, since the above-mentioned input feature map is regarded as the input feature map of global average pooling, after obtaining the input feature map processed by the convolution layer in the neural network, the input feature map of the depth separable convolution is determined according to the input feature map. The specific determination scheme has been introduced in detail in the aforementioned embodiment, so it will not be repeated here. On the basis of obtaining the input feature map of the depth separable convolution, in each channel of the input feature map of the depth separable convolution, each channel is convolved according to the convolution kernel of each channel to obtain the output value of each channel. Since the input feature map of the depth separable convolution and the input feature map of the global average pooling are the same, the convolution calculation process can refer to the above-mentioned convolution calculation process, which will not be repeated here.

[0078] The above is the specific implementation process of using depthwise separable convolution to implement global average pooling operation in the pooling layer of the neural network. It can enable chips that support depthwise separable convolution but do not support global average pooling operation to support global average pooling calculation of any size without adding additional hardware.

[0079] Exemplary Devices

[0080] Figure 6 The figure illustrates a block diagram of a device 600 for generating an instruction sequence of a neural network according to an embodiment of the present application.

[0081] like Figure 6 As shown, the instruction sequence generating device 600 of the neural network of the embodiment of the present application includes: a first determining module 601, which is used to determine the feature map dimension of the depthwise separable convolution in the neural network according to the feature map dimension of the global average pooling layer in the neural network; a second determining module 602, which is used to determine the dimension of the convolution kernel used for the depthwise separable convolution according to the input feature map dimension of the global average pooling layer; a generating module 603, which is used to generate an instruction sequence corresponding to the execution of the neural network according to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, wherein the instruction sequence includes the operation instructions of the global average pooling layer of the neural network.

[0082] In one example, the feature map dimensions include: feature map height, feature map width and feature map channel number; the first determination module is specifically used to: determine the input feature map dimension of the global average pooling layer as the input feature map dimension of the depth-wise separable convolution; determine the output feature map dimension of the global average pooling layer as the output feature map dimension of the depth-wise separable convolution.

[0083] Figure 7 The diagram shows an example block diagram of the second determination module 602 according to an embodiment of the present application. Figure 7As shown, in one example, the second determination module 602 includes: a first determination submodule 701, used to determine the height and width of the convolution kernel based on the input feature map height and input feature map width of the global average pooling layer, respectively; a second determination submodule 702, used to determine the number of convolution kernels according to the number of input feature map channels of the global average pooling layer, so that the number of convolution kernels is the same as the number of input feature map channels; a third determination submodule 703, used to determine the value of the convolution kernel according to the input feature map height and input feature map width of the global average pooling layer.

[0084] In one example, the third determination module is specifically used to determine the product between the input feature map height and the input feature map width of the global average pooling layer; and determine the value of the convolution kernel based on the inverse of the product.

[0085] In one example, the device further includes: a fourth determination module, used to determine the precision supported by the target hardware; and an adjustment module, used to adjust the value of the convolution kernel according to the precision supported by the target hardware.

[0086] In one example, the generating module 603 is further used to generate a parameter file for executing the neural network according to the instruction sequence.

[0087] Based on the same inventive concept and beneficial effects as one or more of the above embodiments, Figure 8 A block diagram of an apparatus 800 for operating a neural network according to an embodiment of the present application is illustrated, wherein the neural network includes an instruction sequence generated according to one or more of the aforementioned embodiments.

[0088] like Figure 8 As shown, the device 800 for running a neural network in an embodiment of the present application includes: an obtaining unit 801, used to obtain an input feature map after processing by a convolution layer in the neural network; wherein the input feature map includes at least the number of channels; a determining unit 802, used to determine the number of convolution kernels according to the number of channels of the input feature map; wherein one channel corresponds to one convolution kernel; a convolution unit 803, used to convolve each channel of the input feature map according to the convolution kernel of each channel to obtain an output value of each channel; wherein the output value of each channel is the same as the output value of the channel obtained by a global average pooling operation.

[0089] In one example, the value of the convolution kernel corresponding to each channel is the same, wherein the value is the inverse of a product, and the product is obtained by multiplying the height and width of the convolution kernel; the height and width of the convolution kernel are the same as the height and width of the input feature map.

[0090] In one example, the convolution unit 803 is specifically used to determine, for each channel, in the height direction, the product of each row in the height direction according to the step size in the height direction when each row in the channel and the corresponding convolution kernel moves; merge the product of each row in the height direction and the coordinate value of each row in the current height direction to obtain the first input parameter of each row in the height direction; in the width direction, determine the product of each column in the width direction according to the step size in the width direction when each column in the channel and the corresponding convolution kernel moves; merge the product of each column in the width direction and the coordinate value of each row in the current width direction to obtain the second input parameter of each row in the width direction; and obtain the output value of the channel by weighted summation of the first input parameter, the second input parameter and the corresponding convolution kernel.

[0091] Exemplary removable device equipment

[0092] Fig. 9 A block diagram of a mobile device according to an embodiment of the present application is illustrated.

[0093] like Fig. 9 As shown, the mobile device (electronic device 10 ) includes one or more processors 11 and a memory 12 .

[0094] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the mobile device to perform desired functions.

[0095] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the distance measurement method of the target object of each embodiment of the present application above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0096] In one example, the mobile device may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0097] For example, when the movable device is the first device or the second device, the input device 13 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, for receiving collected input signals from the first device and the second device.

[0098] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0099] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0100] Of course, to simplify, Fig. 9 Only some of the components in the mobile device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the mobile device may also include any other appropriate components.

[0101] Exemplary computer program products and computer-readable storage media

[0102] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the instruction sequence generation method for a neural network according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0103] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0104] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for generating an instruction sequence for a neural network according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0105] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0106] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0107] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0108] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0109] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0110] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating an instruction sequence of a neural network, wherein the structure of the neural network comprises: A convolution layer, a pooling layer, and a fully connected layer, wherein the pooling layer reuses the processing logic of the convolution layer, uses depthwise separable convolution to implement a global average pooling operation, and the neural network itself does not change at all. The method includes: Determining a feature map dimension of a depthwise separable convolution in the neural network according to a feature map dimension of a global average pooling layer in the neural network; According to the dimension of the input feature map of the global average pooling layer, determine the dimension of the convolution kernel used for the depthwise separable convolution; the dimension of the convolution kernel includes: the height of the convolution kernel, the width of the convolution kernel, the number of channels of the convolution kernel, the value of the convolution kernel and the number of convolution kernels; determine the precision supported by the target hardware; adjust the value of the convolution kernel according to the precision supported by the target hardware; wherein the value of the convolution kernel is determined by: Where J represents the precision supported by the target hardware, H in , W in They represent the height and width of the input feature map of the global average pooling layer, respectively, and k' represents the value of the adjusted convolution kernel; According to the feature map dimension of the depthwise separable convolution and the dimension of the convolution kernel, an instruction sequence for executing the corresponding neural network is generated, and the instruction sequence includes operation instructions of the global average pooling layer of the neural network.

2. The method of claim 1, wherein: The feature map dimensions include: feature map height, feature map width and feature map channel number; Determining the feature map dimension of the depthwise separable convolution in the neural network according to the feature map dimension of the pooling layer in the neural network includes: Determine the input feature map dimension of the global average pooling layer as the input feature map dimension of the depthwise separable convolution; The output feature map dimension of the global average pooling layer is determined as the output feature map dimension of the depth-wise separable convolution.

3. The method of claim 2, wherein determining the dimension of the convolution kernel for depthwise separable convolution according to the dimension of the input feature map of the global average pooling layer comprises: Determine the height and width of the convolution kernel based on the input feature map height and input feature map width of the global average pooling layer; Determining the number of convolution kernels according to the number of input feature map channels of the global average pooling layer so that the number of convolution kernels is the same as the number of input feature map channels; The value of the convolution kernel is determined according to the input feature map height and input feature map width of the global average pooling layer.

4. The method according to claim 3, wherein determining the value of the convolution kernel according to the input feature map height and input feature map width of the global average pooling layer specifically comprises: Determining the product between the input feature map height and the input feature map width of the global average pooling layer; A value of the convolution kernel is determined based on the inverse of the product.

5. A method for operating a neural network, wherein the neural network includes an instruction sequence generated according to any one of claims 1 to 4; the method comprising: Obtaining an input feature map after being processed by a convolutional layer in the neural network; wherein the input feature map includes at least the number of channels; Determine the number of convolution kernels according to the number of channels of the input feature map; wherein one channel corresponds to one convolution kernel; In each channel of the input feature map, convolution is performed on each channel according to the convolution kernel of each channel to obtain an output value of each channel; wherein the output value of each channel is the same as the output value obtained by the global average pooling operation on the channel.

6. The method of claim 5, wherein: The iterating each channel according to the convolution kernel of each channel to obtain the output value of each channel includes: For each channel, in the height direction, the product of each row in the height direction is determined according to each row in the channel and the step length in the height direction when the corresponding convolution kernel moves; the product of each row in the height direction and the coordinate value of each row in the current height direction are combined to obtain the first input parameter of each row in the height direction; In the width direction, the product of each column in the width direction is determined according to the step length in the width direction when each column in the channel and the corresponding convolution kernel move; the product of each column in the width direction and the coordinate value of each row in the current width direction are combined to obtain the second input parameter of each row in the width direction; The output value of the channel is obtained by weighted summing the first input parameter, the second input parameter and the corresponding convolution kernel.

7. A device for generating an instruction sequence of a neural network, the structure of the neural network comprising: Convolutional layer, pooling layer, fully connected layer, the pooling layer reuses the processing logic of the convolutional layer, uses depthwise separable convolution to implement global average pooling operation, and the neural network itself does not change, including: A first determination module is used to determine the feature map dimension of the depthwise separable convolution in the neural network according to the feature map dimension of the global average pooling layer in the neural network; The second determination module is used to determine the dimension of the convolution kernel used for depthwise separable convolution according to the dimension of the input feature map of the global average pooling layer; the dimension of the convolution kernel includes: the height of the convolution kernel, the width of the convolution kernel, the number of channels of the convolution kernel, the value of the convolution kernel and the number of convolution kernels; determine the precision supported by the target hardware; adjust the value of the convolution kernel according to the precision supported by the target hardware; wherein the value of the convolution kernel is determined by: Where J represents the precision supported by the target hardware, H in , W in They represent the height and width of the input feature map of the global average pooling layer, respectively, and k' represents the value of the adjusted convolution kernel; A generation module is used to generate an instruction sequence corresponding to the execution of the neural network according to the feature map dimension of the depth-separable convolution and the dimension of the convolution kernel, wherein the instruction sequence includes operation instructions of the global average pooling layer of the neural network.

8. A device for operating a neural network, wherein the neural network comprises an instruction sequence generated according to any one of claims 1 to 4; comprising: An obtaining unit, used to obtain an input feature map after being processed by a convolutional layer in the neural network; wherein the input feature map at least includes the number of channels; A determination unit, used to determine the number of convolution kernels according to the number of channels of the input feature map; wherein one channel corresponds to one convolution kernel; A convolution unit is used to convolve each channel of the input feature map according to the convolution kernel of each channel to obtain an output value of each channel; wherein the output value of each channel is the same as the output value obtained by the global average pooling operation on the channel.

9. An electronic device, comprising: processor; as well as A memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 6.

Images