A data processing method and device, and an accelerator for a convolutional neural network

Abstract

The application provides a data processing method and device and an accelerator of a convolutional neural network. The method comprises: buffering data with consistent row attributes in data of each target input channel of the convolutional neural network into the same memory to obtain data of each target input channel corresponding to at least one row attribute; obtaining at least one target data from the data of each target input channel corresponding to each row attribute in parallel based on a plurality of convolution kernels to obtain target data corresponding to each target input channel; and distributing the target data corresponding to each target input channel to different multiplier groups in at least one multiplier group so that the multiplier groups perform multiplication operations on the target data corresponding to each target input channel and convolution operators in the convolution kernels to obtain multiplication operation results, and so that adder groups corresponding to the multiplier groups perform addition operations on the multiplication operation results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method, device and accelerator of convolutional neural network. BACKGROUND

[0002] At present, the deep separable convolutional neural network can greatly reduce the operation amount and speed up the calculation to a certain extent without losing accuracy.

[0003] However, when running the deep separable convolutional neural network on the existing convolutional neural network accelerator, the utilization rate of the multiplier in the convolutional neural network accelerator is not high, so how to improve the utilization rate of the multiplier in the convolutional neural network accelerator corresponding to the deep separable convolutional neural network becomes a problem. SUMMARY

[0004] The present application provides the following technical solutions:

[0005] The present application provides a data processing method, comprising:

[0006] Buffering the data with consistent row attributes in the data of each target input channel of the convolutional neural network to the same memory to obtain the data of each target input channel corresponding to at least one row attribute, and the data of each target input channel corresponds to the size of the multiplier group in the at least one multiplier group and the size of the convolution kernel in the plurality of convolution kernels used for separable convolution operation;

[0007] Parallelly obtaining at least one target data from the data of each target input channel corresponding to at least one row attribute based on the plurality of convolution kernels to obtain the target data corresponding to each target input channel;

[0008] At least distributing the target data corresponding to each target input channel to different multiplier groups in the at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain the multiplication operation result, so that the adder group corresponding to the multiplier group performs addition operation on the multiplication operation result.

[0009] The method further comprises:

[0010] Writing the data of each target input channel corresponding to each row attribute into a shift register matrix;

[0011] The parallelly obtaining at least one target data from the data of each target input channel corresponding to at least one row attribute based on the plurality of convolution kernels to obtain the target data corresponding to each target input channel comprises:

[0012] If the size of the convolution kernel supported by the multiplier is not less than the size of the convolution kernel, the data of each target input channel corresponding to the at least one row attribute is obtained from the shift register matrix based on the plurality of convolution kernels at one time, and each target data of each target input channel is obtained.

[0013] The at least target data corresponding to each target input channel is distributed to different multiplier groups in the at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain a multiplication result, and the adder group corresponding to the multiplier group performs addition operation on the multiplication result.

[0014] Each target input channel is grouped to obtain at least one group of target input channels.

[0015] If the multiplier group supports multiplication operation on the target data corresponding to the group of target input channels, the target data corresponding to each target input channel in each group of target input channels is distributed to different multiplier groups in the at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to each target input channel in each group of target input channels and the convolution operator in the convolution kernel to obtain a multiplication result corresponding to each target input channel, and the adder group corresponding to the multiplier group performs addition operation on the multiplication result corresponding to each target input channel.

[0016] The method further comprises:

[0017] If the multiplier group does not support multiplication operation on the target data corresponding to the group of target input channels, a plurality of first target data are obtained from the target data corresponding to each target input channel in each group of target input channels based on the size of the multiplier group, and the first target data corresponding to each group of target input channels are obtained.

[0018] The first target data corresponding to each group of target input channels are distributed to different first multiplier groups in the at least one multiplier group, so that the first multiplier group performs multiplication operation on the first target data corresponding to each group of target input channels and the convolution operator in the convolution kernel to obtain a first multiplication result corresponding to each target input channel.

[0019] The second target data in the target data corresponding to each of the target input channels except the first target data is allocated to a second multiplier group in the at least one multiplier group, so that the second multiplier group performs multiplication operation on the corresponding convolution operator in the convolution kernel and the second target data corresponding to each of the target input channels, to obtain a second multiplication operation result corresponding to each of the target input channels;

[0020] The first multiplication operation result corresponding to each of the target input channels is added by a first adder group corresponding to the first multiplier group, to obtain a first addition operation result, and the first addition operation result and the second multiplication operation result corresponding to each of the target input channels are added, to obtain a second addition operation result.

[0021] The method further comprises:

[0022] If the multiplier group supports multiplication operation on the target data corresponding to the group of target input channels, an adder group corresponding to the multiplier group is switched to a first adder group, and the first adder group is used to perform addition operation on the multiplication operation result corresponding to the multiplier group.

[0023] If the multiplier group does not support multiplication operation on the target data corresponding to the group of target input channels, the adder group corresponding to the multiplier group is switched to a second adder group, and the second adder group is used to perform addition operation on the multiplication operation result corresponding to the multiplier group and the multiplication operation result corresponding to another multiplier group except the multiplier group.

[0024] The first target data corresponding to each of the target input channels in each of the group of target input channels is obtained based on the size of the multiplier group, to obtain the first target data corresponding to each of the group of target input channels, and the number of the first target data corresponding to each of the group of target input channels is consistent with the number of multipliers in the multiplier group.

[0025] The first target data corresponding to each of the target input channels in each of the group of target input channels is obtained based on the size of the multiplier group, to obtain the first target data corresponding to each of the group of target input channels, and the number of the first target data corresponding to each of the group of target input channels is consistent with the number of multipliers in the multiplier group.

[0026] Another aspect of the present application provides a data processing device, comprising:

[0027] a buffer unit, configured to buffer data with consistent row attributes in data of each target input channel of the convolutional neural network into a same memory, to obtain data of each target input channel corresponding to at least one row attribute, the data of each target input channel corresponding to a size of the multiplier group in the at least one multiplier group and a size of the convolution kernel in the plurality of convolution kernels used for separable convolution operation;

[0028] a obtaining unit, configured to obtain at least one target data from the data of each target input channel corresponding to the at least one row attribute in parallel based on the plurality of convolution kernels, to obtain target data corresponding to each target input channel;

[0029] a distribution unit, configured to distribute the target data corresponding to each target input channel to different multiplier groups in the at least one multiplier group, so that the multiplier groups perform multiplication operation on the target data corresponding to the target input channel and a convolution operator in the convolution kernel, to obtain multiplication operation results, so that an adder group corresponding to the multiplier group performs addition operation on the multiplication operation results.

[0030] The third aspect of the present application provides an accelerator of a convolutional neural network, comprising:

[0031] at least one memory, at least one multiplier group and at least one adder group;

[0032] the memory is configured to buffer data with consistent row attributes in data of each target input channel of the convolutional neural network, the data of each target input channel corresponding to a size of the multiplier group in the at least one multiplier group and a size of the convolution kernel in the plurality of convolution kernels used for separable convolution operation;

[0033] the multiplier group is configured to perform multiplication operation on the target data corresponding to the target input channel and a convolution operator in the convolution kernel, to obtain multiplication operation results, the target data corresponding to the target input channel being obtained by obtaining at least one target data from the data of each target input channel corresponding to each row attribute in parallel based on the plurality of convolution kernels;

[0034] the adder group is configured to perform addition operation on multiplication operation results of the multiplier group corresponding to the adder group.

[0035] the at least one adder group comprises at least one first adder group and at least one second adder group;

[0036] the first adder group is configured to perform addition operation on multiplication operation results corresponding to the multiplier group;

[0037] The second adder group is configured to add the multiplication result corresponding to the multiplier group and the multiplication result corresponding to another multiplier group except the multiplier group.

[0038] The number of the at least one memory is consistent with the number of the column weight data of the convolution kernel.

[0039] The capacity of the memory is not less than the data amount corresponding to the data with consistent row attributes in the data of each target input channel of the convolutional neural network.

[0040] The accelerator further comprises a shift register matrix configured to store the data of each target input channel corresponding to each row attribute.

[0041] In the present application, the data with consistent row attributes in the data of each target input channel of the convolutional neural network is buffered in the same memory to obtain the data of each target input channel corresponding to each row attribute, at least one target data is obtained from the data of each target input channel corresponding to each row attribute based on the convolution kernel for separable convolution operation, the target data corresponding to each target input channel is obtained, the target data corresponding to the target input channel is realized to contain at least one target data of the target input channel corresponding to each row attribute, and at least the target data corresponding to each target input channel is distributed to different multiplier groups in at least one multiplier group. The multiplier group can perform multiplication operation on at least one target data of the target input channel corresponding to each row attribute, and the utilization rate of the multiplier group is improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a flow diagram of a data processing method provided by embodiment 1 of the present application

[0044] Figure 2 is a schematic diagram of an implementation scenario of the data processing method provided by embodiment 1 of the present application

[0045] Figure 3 is another schematic diagram of an implementation scenario of the data processing method provided by embodiment 1 of the present application

[0046] Figure 4 is still another schematic diagram of an implementation scenario of the data processing method provided by embodiment 1 of the present application

[0047] Figure 5 is another implementation scenario of the data processing method provided by Embodiment 1 of the present application;

[0048] Figure 6 is a data storage scenario provided by Embodiment 1 of the present application;

[0049] Figure 7 is another implementation scenario of the data processing method provided by Embodiment 1 of the present application;

[0050] Figure 8 is another implementation scenario of the data processing method provided by Embodiment 1 of the present application;

[0051] Figure 9 is a flowchart of a data processing method provided by Embodiment 2 of the present application;

[0052] Figure 10 is an implementation scenario of the data processing method provided by Embodiment 2 of the present application;

[0053] Figure 11 is an implementation scenario of the data processing method provided by Embodiment 3 of the present application;

[0054] Figure 12 is an implementation scenario of the data processing method provided by Embodiment 4 of the present application;

[0055] Figure 13 is a structure diagram of an adder group provided by the present application;

[0056] Figure 14 is another structure diagram of an adder group provided by the present application;

[0057] Figure 15 is a structure diagram of an accelerator of a convolutional neural network provided by the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0060] Reference Figure 1A flowchart of a data processing method provided for Embodiment 1 of the present application is shown in FIG. 1, which can include but is not limited to the following steps: Figure 1

[0061] In step S101, data with consistent row attributes in each target input channel of the convolutional neural network is buffered into the same memory to obtain data of each target input channel corresponding to at least one row attribute.

[0062] In this embodiment, based on the size of the multiplier group in the at least one multiplier group and the size of the convolution kernel in the plurality of convolution kernels used for the separable convolution operation, data of each target input channel of the convolutional neural network can be obtained from the data of each input channel of the convolutional neural network, the number of each target input channel corresponds to the size of the multiplier group in the at least one multiplier group, and the data of each target input channel corresponds to the size of the multiplier group in the at least one multiplier group and the size of the convolution kernel in the plurality of convolution kernels used for the separable convolution operation. For example, as shown in FIG. 2, the input channels of the convolutional neural network include C input channels, each input channel includes W*H input data, and if the size of the multiplier group is that the multiplier group includes AC multipliers and the size of the convolution kernel is R*S, based on AC and R*S, AC target input channel input data can be obtained from the W*H*C input data of the convolutional neural network, and the data of each target input channel of the convolutional neural network is shown in FIG. 3, i.e., R*S*AC input data. Figure 2 Figure 2

[0063] Of course, in this embodiment, the data of each target input channel of the convolutional neural network is not limited to being obtained from the data of each input channel of the convolutional neural network based on the size of the multiplier group in the at least one multiplier group and the size of the convolution kernel in the plurality of convolution kernels used for the separable convolution operation. It can also be obtained from the data of each input channel of the convolutional neural network based on the size of the multiplier group in the at least one multiplier group. For example, as shown in FIG. 4, the input channels of the convolutional neural network include C input channels, each input channel includes W*H input data, and if the size of the multiplier group is that the multiplier group includes AC multipliers, based on AC, AC target input channel input data can be obtained from the W*H*C three-dimensional input data of the convolutional neural network, and the data of each target input channel of the convolutional neural network is shown in FIG. 5, i.e., W*H*AC input data. Figure 3 Figure 3

[0064] ​​​​​It can be understood that the number of at least one multiplier group and the number of multipliers in the multiplier group can be set as needed, and is not limited in the present application. The number of at least one multiplier group and the number of multipliers in the multiplier group can be the same or different.

[0065] The data of the target input channel of the convolutional neural network can be, but is not limited to, arranged in a rectangular array. In the embodiment of arranging the data of the target input channel of the convolutional neural network in a rectangular array, the row attribute can be understood as the attribute corresponding to each row in the rectangular array.

[0066] In the present embodiment, the number of memories is not limited. The number of memories can be consistent with the number of a column of input data in the rectangular array or consistent with the number of a column of weight data in the convolution kernel.

[0067] The number of memories is consistent with the number of a column of weight data in the convolution kernel (i.e., the number of rows of the convolution kernel), which can ensure that each memory corresponds to a row attribute, to meet the requirement of buffering the data of each target input channel corresponding to each row attribute, and can also ensure the utilization rate of the memory.

[0068] The capacity of the memory is not less than the data amount corresponding to the data with consistent row attributes in the data of each target input channel of the convolutional neural network.

[0069] The capacity of the memory is not less than the data amount corresponding to the data with consistent row attributes in the data of each target input channel of the convolutional neural network, which can ensure that the memory can buffer the data of each target input channel corresponding to the row attribute for at least one multiplier group, to support the multiplication operation of at least one multiplier group on the data of each target input channel corresponding to the row attribute.

[0070] In the present embodiment, the memory is used to buffer the data with consistent row attributes in the data of each target input channel of the convolutional neural network. The memory can include, but is not limited to, SRAM (Static Random-Access Memory).

[0071] It can be understood that the number of a column of input data in the rectangular array can be consistent with the number of a column of weight data in the convolution kernel, or the number of a column of input data in the rectangular array can be greater than the number of a column of weight data in the convolution kernel.

[0072] In the embodiment of the number of memories being consistent with the number of a column of weight data in the convolution kernel, the number of at least one row attribute is the same as the number of a column of weight data in the convolution kernel.

[0073] This implementation method, based on the size of the multiplier group in at least one multiplier group and the size of the convolution kernels in multiple convolution kernels used for separable convolution operations, obtains the data of each target input channel of the convolutional neural network from the data of each input channel. For example, if the convolution kernel contains 3*3 weight data, the number of memories is 3, the multiplier group includes 8 multipliers, the data of each input channel of the convolutional neural network is 4*4*16 input data, and the data of each target input channel of the convolutional neural network is as follows... Figure 4 The 3*3*8 input data shown in part (a) can be represented as h1 for the first row, h2 for the second row, and h3 for the third row.

[0074] like Figure 4 As shown in section (b), the data with consistent row attributes in the 3*3*8 input data are buffered into the same memory to obtain the data of each target input channel corresponding to the three row attributes. That is, the data at positions h11, h12, and h13 in the first row of each input channel in the 3*3*8 input data are buffered into the first memory; the data at positions h21, h22, and h23 in the second row of each target input channel in the 3*3*8 input data are buffered into the second memory; and the data at positions h31, h32, and h33 in the third row of each target input channel in the 3*3*8 input data are buffered into the third memory.

[0075] This implementation method, based on the size of the multiplier group in at least one multiplier group, extracts the data of each target input channel of the convolutional neural network from the data of each input channel. For example, if the convolutional kernel contains 3*3 weights, the number of memories is 3, the multiplier group includes 8 multipliers, the data of each input channel of the convolutional neural network is 4*4*16 input data, and the data of each target input channel of the convolutional neural network is as follows... Figure 5 The 4*4*8 input data shown in part (a) can be represented as h1 for the first row, h2 for the second row, h3 for the third row, and h4 for the fourth row.

[0076] The data of the same row attribute in the 4*4*8 input data is buffered in the same memory, and the data of each target input channel corresponding to the three row attributes is obtained. The data at positions h11, h12, h13 and h14 in the first row of each input channel in the 4*4*8 input data is buffered in the first memory, the data at positions h21, h22, h23 and h24 in the second row of each target input channel in the 4*4*8 input data is buffered in the second memory, and the data at positions h31, h32, h33 and h34 in the third row of each target input channel in the 4*4*8 input data is buffered in the third memory.

[0077] It can be understood that, as shown in part (a) of Figure 5 Although the data amount of the data buffered in the at least one memory is greater than the data amount of the data corresponding to the size of the convolution kernel, it does not affect the reading of the data from the memories based on the plurality of convolution kernels.

[0078] It should be noted that, for the embodiment in which the number of corresponding memories is 3, the data of each target input channel corresponding to the row attribute h4 is not currently buffered, and with the execution of the separable convolution operation, the data of each target input channel corresponding to the row attribute h4 can be buffered in the first memory to replace the data of each target input channel corresponding to the row attribute h1 to participate in the separable convolution operation.

[0079] It should be noted that, Figure 4 and Figure 5 The logical arrangement of the data of each target input channel corresponding to each row attribute as shown in part (b) does not represent the physical storage manner in the memory.

[0080] In step S102, based on the plurality of convolution kernels, at least one target data is obtained from the data of each target input channel corresponding to each row attribute in parallel, and the target data corresponding to each target input channel is obtained.

[0081] For the embodiment in which the data corresponding to each target input channel corresponds to the size of the convolution kernel in the plurality of convolution kernels and the size of the multiplier group in the at least one multiplier group, based on the plurality of convolution kernels, at least one target data is directly obtained from the data of each target input channel corresponding to each row attribute in parallel, and the target data corresponding to each target input channel is obtained.

[0082] In this embodiment, each memory can buffer the data of each target input channel corresponding to each row attribute, and the starting addresses of the data of each target input channel corresponding to each row attribute buffered in each memory are consistent, and the data width of each memory corresponds to the size of the multiplier group. Based on the above implementation of the corresponding memory, since the data amount of the data of each target input channel corresponding to each row attribute is consistent, the data corresponding to the same address can be read from each memory in parallel based on multiple convolution kernels, and at least one target data is obtained from the data of each target input channel corresponding to each row attribute based on multiple convolution kernels in parallel. For example, if the data of each target input channel of the convolutional neural network is 3*3*8 input data as shown in part (a) of FIG. 8, the data width of the memory is 8, as shown in part (b) of FIG. 8, the addresses of the data of each target input channel corresponding to the position h11 in the first memory are 0-7 respectively, the addresses of the data of each target input channel corresponding to the position h12 are 8-15 respectively, and the addresses of the data of each target input channel corresponding to the position h13 are 16-23 respectively. Figure 4 Figure 6

[0083]

[0084]

[0085]

[0086]

[0087] ​​​​​​The data corresponding to the 16-23 address pairs can be read from the first memory, the second memory and the third memory in parallel to obtain the target data corresponding to each target input channel (i.e., the data corresponding to the h13 position of each target input channel, the data corresponding to the h23 position of each target input channel and the data of the h33 position of each target input channel).

[0088] Based on the plurality of convolution kernels, at least one target data is obtained from the data of each target input channel corresponding to each row attribute in parallel to obtain the target data corresponding to each target input channel. It can be understood that, based on the convolution kernel corresponding to each target input channel in the plurality of convolution kernels, at least one target data is obtained from the data of each target input channel corresponding to each row attribute in parallel to obtain the target data corresponding to each target input channel.

[0089] It can be understood that, in the separable convolution operation, the number of convolution kernels is the same as the number of input channels of the convolutional neural network, and each input channel corresponds to a convolution kernel. Correspondingly, each target input channel corresponds to a convolution kernel, and the convolution kernels corresponding to each target input channel are different.

[0090] In the embodiment in which the data corresponding to each target input channel corresponds to the size of the multiplier group in at least one multiplier group, the step can include but is not limited to:

[0091] S1021, based on the plurality of convolution kernels, at least one target data is obtained from the data set corresponding to the convolution kernel in the data of each target input channel corresponding to each row attribute in parallel to obtain the target data corresponding to each target input channel.

[0092] It should be noted that there can be multiple data sets corresponding to the convolution kernel in the data of each target input channel corresponding to each row attribute. If there are multiple data sets, step S1021 is performed respectively for each data set corresponding to the convolution kernel.

[0093] The target data corresponding to the target input channel includes at least one target data of the target input channel corresponding to each row attribute. The number of target data corresponding to each target input channel is at least not less than the number of one column of weight data of the convolution kernel. For example, if the size of the convolution kernel used in the separable convolution operation is 3*3, the data of the h11 position, the h12 position and the h13 position of each target input channel corresponding to the row attribute h1 is shown in part (b) of FIG. 1, the data of the h21 position, the h22 position and the h23 position of each target input channel corresponding to the row attribute h2, and the data of the h31 position, the h32 position and the h33 position of each target input channel corresponding to the row attribute h3; or, as shown in part (c) of FIG. 1, the data of the h11 position, the h12 position and the h13 position of each target input channel corresponding to the row attribute h1, the data of the h21 position, the h22 position and the h23 position of each target input channel corresponding to the row attribute h2, and the data of the h31 position, the h32 position and the h33 position of each target input channel corresponding to the row attribute h3. Figure 4 Figure 5 ​The data of the h11 position, the data of the h12 position, the data of the h13 position and the data of the h14 position of each target input channel corresponding to the row attribute h1 shown in the part (b), the data of the h21 position, the data of the h22 position, the data of the h23 position and the data of the h24 position of each target input channel corresponding to the row attribute h2, and the data of the h31 position, the data of the h32 position, the data of the h33 position and the data of the h24 position of each target input channel corresponding to the row attribute h3;

[0094] The target data corresponding to the target input channel includes the data of the h11 position of the target input channel corresponding to the row attribute h1, the data of the h21 position of the target input channel corresponding to the row attribute h2 and the data of the h31 position of the target input channel corresponding to the row attribute h3.

[0095] Or, the data of the h12 position of the target input channel corresponding to the row attribute h1, the data of the h22 position of the target input channel corresponding to the row attribute h2 and the data of the h32 position of the target input channel corresponding to the row attribute h3.

[0096] Or, the data of the h13 position of the target input channel corresponding to the row attribute h1, the data of the h23 position of the target input channel corresponding to the row attribute h2 and the data of the h33 position of the target input channel corresponding to the row attribute h3.

[0097] If there are multiple row attributes, multiple convolution kernels are used to obtain at least one target data from the data of each target input channel corresponding to each row attribute in parallel, so that multiple target data corresponding to each target input channel can be obtained using one clock.

[0098] In step S103, at least the target data corresponding to each target input channel is distributed to different multiplier groups in at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain the multiplication result, and the adder group corresponding to the multiplier group performs addition operation on the multiplication result.

[0099] If the number of at least one multiplication group is less than the number of target input channels, at least the target data corresponding to each target input channel is distributed to different multiplier groups in at least one multiplier group, and one multiplier group may be distributed to multiple target data corresponding to multiple target input channels. In the case that one multiplier group is distributed to multiple target data corresponding to multiple target input channels, the multiplier group respectively performs multiplication operation on the target data corresponding to each target input channel and the convolution operator corresponding to the target input channel.

[0100] If the number of at least one multiplication group is not less than the number of each target input channel, at least the target data corresponding to each target input channel is allocated to different multiplication groups in at least one multiplication group, which can ensure that each target input channel corresponds to a multiplication group. For example, if there are AK multiplication groups, each multiplication group includes AC multipliers, AK = AC, the input data of the convolutional neural network is three-dimensional input data of W*H*C, and the size of the convolution kernel is R*S, as shown in Figure 7 The data corresponding to the h11 position of the target input channel d1, the data corresponding to the h21 position, and the data corresponding to the h31 position are allocated to the multiplication group X1, the data corresponding to the h11 position of the target input channel d2, the data corresponding to the h21 position, and the data corresponding to the h31 position are allocated to the multiplication group X2, …, the data corresponding to the h11 position of the target input channel d AC , the data corresponding to the h21 position, and the data corresponding to the h31 position are allocated to the multiplication group X AK .

[0101] Each multiplication group can perform multiplication operation on R data, and the utilization rate of the multiplication group is R / AC.

[0102] The adder group corresponding to the multiplication group performs addition operation on the multiplication result, that is, the adder group J1 performs addition operation on the multiplication result of the multiplication group X1, the adder group J2 performs addition operation on the multiplication result of the multiplication group X2, …, and the adder group J AK performs addition operation on the multiplication result of the multiplication group X AK , which can improve the efficiency of separable convolution operation.

[0103] It should be noted that the data of each target input channel of the convolutional neural network is a data set of the data of each input channel of the convolutional neural network, and the data of each input channel of the convolutional neural network may include multiple data sets. If the data of each input channel of the convolutional neural network includes multiple data sets, for each data set, the above steps S101-S103 are executed respectively.

[0104] In the embodiment, data with consistent row attributes in the data of each target input channel of the convolutional neural network is buffered in the same memory to obtain the data of each target input channel corresponding to at least one row attribute, at least one target data is obtained from the data of each target input channel corresponding to at least one row attribute based on the convolution kernel for separable convolution operation, the target data corresponding to each target input channel is obtained, the target data corresponding to the target input channel contains at least one target data of the target input channel corresponding to each row attribute, and at least the target data corresponding to each target input channel is distributed to different multiplier groups in at least one multiplier group. The multiplier group can perform multiplication operation on at least one target data of the target input channel corresponding to each row attribute, and the utilization rate of the multiplier group is improved. For example, if there are AK multiplier groups, each multiplier group includes AC multipliers, AK = AC, the input data of the convolutional neural network is three-dimensional input data with a size of W*H*C, and the size of the convolution kernel is R*S.

[0105] In the case of buffering the data of each target input channel of the convolutional neural network in one memory, one clock can be used to read one data from the memory, as shown in Figure 8 The data of each target input channel of the convolutional neural network is distributed to different multiplier groups in at least one multiplier group, and the utilization rate of the multiplier group is 1 / AC; in the case of buffering data with consistent row attributes in the data of each target input channel of the convolutional neural network in the same memory, one clock can be used to read at least one data from multiple memories in parallel to obtain the target data corresponding to each target input channel, as shown in Figure 7 The target data corresponding to each target input channel is distributed to different multipliers, and the utilization rate of the multiplier group is R / AC, and the utilization rate of the multiplier is improved.

[0106] As another optional embodiment of the present application, refer to Figure 9 A flowchart of a data processing method provided in Embodiment 2 of the present application is shown in the figure. The embodiment mainly extends the data processing method described in Embodiment 1, and the method can include but is not limited to the following steps:

[0107] In step S201, data with consistent row attributes in the data of each target input channel of the convolutional neural network is buffered in the same memory to obtain the data of each target input channel corresponding to at least one row attribute, and the data of each target input channel corresponds to the size of the multiplier group in at least one multiplier group and the size of the convolution kernel in the plurality of convolution kernels for separable convolution operation.

[0108] For a detailed description of step S201, please refer to the relevant description of step S101 in Example 1, which will not be repeated here.

[0109] Step S202: Write the data of each target input channel corresponding to each row attribute into the shift register matrix.

[0110] The data of each target input channel corresponding to each row attribute is written into the shift register matrix. The shift register matrix can store the data of each target input channel corresponding to at least one row attribute.

[0111] Step S203: If the size of the convolution kernel supported by the multiplier is not less than the size of the convolution kernel, the data of each target input channel corresponding to at least one row attribute is obtained from the shift register matrix at one time based on multiple convolution kernels, so as to obtain the target data of each target input channel.

[0112] If the size of the convolution kernel supported by the multiplier is not less than the size of the convolution kernel, the number of data points that the multiplier can perform multiplication operations on is not less than the number of weight data points contained in the convolution kernel. For example, if the size of the convolution kernel held by the multiplier is 4*4, the size of the convolution kernel used for separable convolution operations is 3*3, and the number of weight data points contained in the convolution kernel used for separable convolution operations is 9, then the number of data points that the multiplier can perform multiplication operations on is 16, which is not less than 9.

[0113] The process of obtaining target data for each target input channel by acquiring data from at least one row attribute corresponding to each target input channel at once from the shift register matrix using multiple convolution kernels can be understood as: acquiring data from at least one row attribute corresponding to each target input channel at once from the shift register matrix using multiple convolution kernels corresponding to each target input channel, thus obtaining target data for each target input channel.

[0114] Each target data in the target input channel contains data corresponding to at least one row attribute of the target input channel. For example, if at least one multiplier group has AK multipliers, and each multiplier group includes 16 multipliers (i.e., one implementation of AC), the input data of the convolutional neural network is 4*4*64 (i.e., W*H*C) input data, and the size of the convolutional kernel is 3*3 (i.e., one implementation of R*S), such as... Figure 10As shown, the data for each target input channel corresponding to at least one row attribute is the data at positions h11, h12, and h13 of each target input channel corresponding to row attribute h1; the data at positions h21, h22, and h23 of each target input channel corresponding to row attribute h2; and the data at positions h31, h32, and h33 of each target input channel corresponding to row attribute h3. Each target data for a target input channel includes the data at positions h11, h12, and h13 of the target input channel corresponding to row attribute h1; the data at positions h21, h22, and h23 of the target input channel corresponding to row attribute h2; and the data at positions h31, h32, and h33 of the target input channel corresponding to row attribute h3.

[0115] In this embodiment, by using multiple convolution kernels to obtain data from at least one row attribute corresponding to each target input channel from the shift register matrix at once, it is possible to obtain data from at least one row attribute corresponding to each target input channel using only one clock cycle.

[0116] Step S203 is a specific implementation of step S102 in Example 1.

[0117] Step S204: At least the target data corresponding to each target input channel is assigned to different multiplier groups in at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain the multiplication operation result, so that the adder group corresponding to the multiplier group performs addition operation on the multiplication operation result.

[0118] For a detailed description of step S204, please refer to the relevant description of step S103 in Example 1, which will not be repeated here.

[0119] For example, such as Figure 10 As shown, each target data in the target input channel includes data at positions h11, h12, and h13 corresponding to row attribute h1; data at positions h21, h22, and h23 corresponding to row attribute h2; and data at positions h31, h32, and h33 corresponding to row attribute h3. Figure 10As shown, the data corresponding to the h11 position of the target input channel d1, the data of the h12 position, the data of the h13 position, the data corresponding to the h21 position, the data of the h22 position, the data of the h23 position, the data of the h31 position, the data of the h32 position and the data of the h33 position are allocated to the multiplier group X1, the data corresponding to the h11 position of the target input channel d2, the data of the h12 position, the data of the h13 position, the data corresponding to the h21 position, the data of the h22 position, the data of the h23 position, the data of the h31 position, the data of the h32 position and the data of the h33 position are allocated to the multiplier group X2, and so on.

[0120] Each multiplier group can perform multiplication operation on R*S data, and the utilization rate of the multiplier group is R*S / AC.

[0121] The adder group corresponding to the multiplier group performs addition operation on the multiplication operation result, and the efficiency of the separable convolution operation can be improved.

[0122] In this embodiment, the data with the same row attribute in the data of each target input channel of the convolutional neural network is buffered in the same memory, the data of each target input channel corresponding to each row attribute is obtained, the data of each target input channel corresponding to each row attribute is written into the shift register matrix, if the size of the convolution kernel supported by the multiplier is not less than the size of the convolution kernel, the data of each target input channel corresponding to at least one row attribute is obtained from the shift register matrix at one time based on multiple convolution kernels, the target data of each target input channel is obtained, the target data of each target input channel contains the data of the target input channel corresponding to at least one row attribute, at least the target data corresponding to each target input channel is allocated to different multiplier groups in at least one multiplier group, and the multiplier group can perform multiplication operation on the data of the target input channel corresponding to at least one row attribute, thereby improving the utilization rate of the multiplier group.

[0123] As another optional embodiment of the present application, the present embodiment mainly provides a data processing method for the embodiment 3 of the present application, and the step S103 in the embodiment 1 or the step S204 in the embodiment 2 can include but is not limited to the following steps.

[0124] S11, grouping each target input channel to obtain at least one group of target input channels.

[0125] In this embodiment, the grouping manner of each target input channel is not limited. For example, each target input channel can be grouped in a manner of two target input channels as a group, at least one group of target input channels is obtained, and each group of target input channels contains two target input channels.

[0126] Among them, the target data corresponding to a set of target input channels includes the target data corresponding to each target input channel in the set of target input channels.

[0127] S12. If the multiplier group supports multiplication of target data corresponding to a set of target input channels, the target data corresponding to each target input channel in at least one set of target input channels is assigned to different multiplier groups in at least one multiplier group, so that the multiplier group performs multiplication of the target data corresponding to each target input channel in a set of target input channels and the convolution operator in the convolution kernel to obtain the multiplication result corresponding to each target input channel, so that the adder group corresponding to the multiplier group performs addition of the multiplication result corresponding to each target input channel.

[0128] The target data corresponding to each target input channel is the implementation method obtained through step S102 in Example 1. The target data corresponding to each target input channel in a set of target input channels allocated to the multiplier group is at least one target data of the target input channel corresponding to each row attribute. For example, if there are AK multiplier groups, each multiplier group includes 16 multipliers (i.e., one implementation of AC), the input data of the convolutional neural network is 4*4*64 (i.e., W*H*C) input data, and the size of the convolutional kernel is 3*3 (i.e., one implementation of R*S);

[0129] like Figure 11 As shown, the target data corresponding to the target input channel is one target data of the target input channel corresponding to each row attribute (e.g., the data at position h11 of the target input channel corresponding to row attribute h1, the data at position h21 of the target input channel corresponding to row attribute h2, and the data at position h31 of the target input channel corresponding to row attribute h3).

[0130] The 16 target input channels are divided into 8 groups (i.e., a specific implementation of dividing AC target input channels into E groups), such as... Figure 11 As shown, the data corresponding to positions h11, h21, and h31 of target input channel d1 and the data corresponding to positions h11, h21, and h31 of target input channel d2 are assigned to multiplier group X1, and the data corresponding to positions h11, h21, and h31 of target input channel d3 and the data corresponding to positions h11, h21, and h31 of target input channel d4 are assigned to multiplier group X2, ...;

[0131] Each multiplier group can respectively perform multiplication operation on R*E data, and the utilization rate of the multiplier group is R*E / AC. Compared with the embodiment that each multiplier group can respectively perform multiplication operation on R data, the utilization rate of the multiplier group is further improved. When AC=64, E=8, and R=3, 5, 7, the utilization rate of the multiplier group is further improved to 8*3 / 64, 8*5 / 64, and 8*7 / 64, i.e., 37.5%, 62.5%, and 87.5%.

[0132] It can be understood that, in the case of buffering the data of each target input channel of the convolutional neural network in one memory, using one clock to read one data from the data of each target input channel of the convolutional neural network in the memory, grouping the target input channels to obtain at least one group of target input channels, and the data corresponding to each target input channel in one group of target input channels is only one, for example, the AC target input channels are divided into E groups, each multiplier group can respectively perform multiplication operation on 1*E data, and the utilization rate of the multiplier group is E / AC. In the embodiment of buffering the data with consistent row attributes of each target input channel of the convolutional neural network in the same memory, the utilization rate of the corresponding multiplier group is R*E / AC, and the utilization rate of the multiplier group is improved.

[0133] The target data corresponding to each target input channel is the embodiment obtained by step S203 in embodiment 2, and the target data corresponding to each target input channel in the group of target input channels to which the multiplier group is assigned is the data of at least one target input channel with consistent row attributes. For example, if there are AK multiplier groups, each multiplier group includes 24 (i.e., one embodiment of AC) multipliers, the input data of the convolutional neural network is 4*4*64 (i.e., W*H*C) input data, and the size of the convolution kernel is 3*3 (i.e., one embodiment of R*S);

[0134] The target data corresponding to each target input channel has 3*3;

[0135] The 24 target input channels are divided into 12 groups (i.e., one specific embodiment of dividing the AC target input channels into E groups), and the target data corresponding to each group of target input channels has 2*3*3 (i.e., one specific embodiment of E*R*S);

[0136] Each multiplier group can respectively perform multiplication operation on E*R*S data, and the utilization rate of the multiplier group is E*R*S / AC. Compared with the embodiment that each multiplier group can respectively perform multiplication operation on R*E data, the utilization rate of the multiplier group is further improved.

[0137] As another optional embodiment of the present application, the data processing method provided in Embodiment 4 of the present application is mainly an extension of the data processing method provided in Embodiment 3 of the present application. The step S103 in Embodiment 1 or the step S204 in Embodiment 2 can include but is not limited to the following steps:

[0138] In step S21, the target input channels are grouped to obtain at least one group of target input channels.

[0139] In step S22, if the multiplier group supports multiplication operation on the target data corresponding to a group of target input channels, the target data corresponding to each group of target input channels in the at least one group of target input channels is distributed to different multiplier groups in the at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to each target input channel in the group of target input channels and the convolution operator in the convolution kernel to obtain the multiplication operation result corresponding to each target input channel, and the adder group corresponding to the multiplier group performs addition operation on the multiplication operation result corresponding to each target input channel.

[0140] The detailed processes of steps S21-S22 can be referred to the related description of steps S11-S12 in Embodiment 3, which will not be repeated here.

[0141] In step S23, if the multiplier group does not support multiplication operation on the target data corresponding to a group of target input channels, a plurality of first target data is obtained from the target data corresponding to each target input channel in each group of target input channels based on the size of the multiplier group, to obtain the first target data corresponding to each group of target input channels.

[0142] In the present embodiment, the number of the first target data corresponding to each group of target input channels can but is not limited to correspond to the size of the multiplier group, i.e., the number of the first target data corresponding to each group of target input channels can but is not limited to be consistent with the number of the multipliers in the multiplier group.

[0143] The target data corresponding to each target input channel is obtained by the implementation of step S203 in Embodiment 2. For example, if the at least one multiplier group has 9 (i.e., an implementation of AK), each multiplier group includes 16 (i.e., an implementation of AC) multipliers, the input data of the convolution neural network is 4*4*64 (i.e., W*H*C) input data, and the size of the convolution kernel is 3*3 (i.e., an implementation of R*S).

[0144] For example, as shown in FIG. 6, the target data corresponding to each target input channel is obtained by the implementation of step S203 in Embodiment 2. For example, if the at least one multiplier group has 9 (i.e., an implementation of AK), each multiplier group includes 16 (i.e., an implementation of AC) multipliers, the input data of the convolution neural network is 4*4*64 (i.e., W*H*C) input data, and the size of the convolution kernel is 3*3 (i.e., an implementation of R*S). Figure 12As shown, each target data of the target input channel includes the data at position h11, position h12 and position h13 of the target input channel corresponding to row attribute h1, the data at position h21, position h22 and position h23 of the target input channel corresponding to row attribute h2, and the data at position h31, position h32 and position h33 of the target input channel corresponding to row attribute h3;

[0145] The 16 target input channels are divided into 8 groups (that is, a specific implementation of dividing AC target input channels into E groups), and each group of target input channels corresponds to 2*3*3 target data.

[0146] like Figure 12 As shown, the multiplier group includes 16 multipliers. It extracts the data corresponding to positions h11-h32 (i.e., the first target data) from the data corresponding to positions h11-h33 of target input channel d1 and target input channel d2, respectively; it also extracts the data corresponding to positions h11-h32 (i.e., the first target data) from the data corresponding to positions h11-h33 of target input channel d3 and target input channel d4, respectively; ... from the data corresponding to positions h11-h33 of target input channel d4... 15 The data corresponding to positions h11-h33 and the target input channel d 16 The data corresponding to positions h11-h32 (i.e., the first target data) are obtained from the data corresponding to positions h11-h33 respectively;

[0147] The number of first target data corresponding to each target input channel is 16.

[0148] Step S24: Assign the first target data corresponding to each group of target input channels to different first multiplier groups in at least one multiplier group, so that the first multiplier group performs multiplication operation on the first target data corresponding to a group of target input channels and the convolution operator in the convolution kernel to obtain the first multiplication operation result corresponding to each target input channel.

[0149] For example, such as Figure 12 As shown, the data corresponding to positions h11-h32 of target input channel d1 and target input channel d2 are assigned to multiplier group X1; the data corresponding to positions h11-h32 of target input channel d3 and target input channel d4 are assigned to multiplier group X2; ... the data corresponding to positions h11-h32 of target input channel d4 are assigned to multiplier group X2; ... the data corresponding to positions h11-h32 of target input channel d4 are assigned to multiplier group X2; ... the data corresponding to positions h11-h32 of target input channel d2 are assigned to multiplier group X2; ... the data corresponding to positions h11-h32 of target input channel d3 and target input channel d4 ... 15 The data corresponding to positions h11-h32 and the target input channel d 16 The data corresponding to positions h11-h32 is assigned to multiplier group X8.

[0150] In this embodiment, the number of first target data corresponding to each group of target input channels may be, but is not limited to, the same as the number of multipliers contained in the multiplier group, and the utilization rate of each first multiplier group can reach AC / AC.

[0151] Step S25: Assign the second target data (excluding the first target data) in the target data corresponding to each target input channel to the second multiplier group in at least one multiplier group, so that the second multiplier group performs multiplication operation on the second target data corresponding to each target input channel and the corresponding convolution operator in the convolution kernel to obtain the second multiplication operation result corresponding to each target input channel.

[0152] For example, the data corresponding to position h33 of target input channel d1 and the data corresponding to position h33 of target input channel d2 are the second target data corresponding to target input channels d1 and d2, respectively; ... target input channel d 15 The data corresponding to position h33 and the target input channel d 16 The data corresponding to position h33 is the target input channel d. 15 and target input channel d 16 The corresponding second target data.

[0153] like Figure 12 As shown, the data corresponding to position h33 of target input channel d1, the data corresponding to position h33 of target input channel d2, ..., the data of target input channel d 16 The data corresponding to position h33 is assigned to multiplier group X9.

[0154] like Figure 12 As shown, the utilization rate of the second multiplier group is AC / AC.

[0155] It should be noted that, Figure 12 This is merely one example and is not intended as a limitation on at least one set of multipliers.

[0156] Step S26: Based on the adder group corresponding to the first multiplication group, perform addition operation on the first multiplication result corresponding to each target input channel to obtain the first addition operation result. Then, perform addition operation on the first addition operation result and the second multiplication operation result corresponding to each target input channel to obtain the second addition operation result.

[0157] In this embodiment, if the multiplier group does not support multiplication operation on the target data corresponding to a group of target input channels, the first multiplier group can be set based on the first multiplier group and the second multiplier group, but not limited thereto. The obtained adder group corresponding to the first multiplier group can perform addition operation on the first multiplication result corresponding to each target input channel to obtain a first addition result. The first addition result corresponding to each target input channel and the second multiplication result are added to obtain a second addition result.

[0158] For example, as shown in FIG. 8, the adder group J1 corresponding to the multiplier group X1 can perform addition operation on the first multiplication result corresponding to the target input channels d1 and d2 obtained by the multiplier group X1 to obtain the first addition result corresponding to the target input channels d1 and d2. The first addition result corresponding to the target input channel d1 and the second multiplication result corresponding to the target input channel d1 obtained by the multiplier group X9 are added to obtain the second addition result corresponding to the target input channel d1. The first addition result corresponding to the target input channel d2 and the second multiplication result corresponding to the target input channel d2 obtained by the multiplier group X9 are added to obtain the second addition result corresponding to the target input channel d2. Figure 12 The addition operation mode of the adder groups corresponding to the multiplier groups X2-X8 is the same as that of the adder group J1, which will not be described herein again.

[0159] In this embodiment, if the multiplier group does not support multiplication operation on the target data corresponding to a group of target input channels, the first multiplier group can be set based on the first multiplier group and the second multiplier group, but not limited thereto. The obtained adder group corresponding to the first multiplier group can perform addition operation on the first multiplication result corresponding to each target input channel to obtain a first addition result. The first addition result corresponding to each target input channel and the second multiplication result are added to obtain a second addition result.

[0160] On the basis of the improved utilization rates of the first multiplier group and the second multiplier group, the overall utilization rate of at least one multiplier group is improved.

[0161] As another optional embodiment of the present application, the data processing method provided in Embodiment 5 is mainly an extension of the data processing method provided in Embodiment 4. The step S103 in Embodiment 1 or the step S204 in Embodiment 2 can include but is not limited to the following steps:

[0162]

[0163] ​Step S31: Group the target input channels to obtain at least one group of target input channels.

[0164] For a detailed description of step S31, please refer to the relevant description of step S21 in Example 4, which will not be repeated here.

[0165] Step S32: If the multiplier group supports multiplication of target data corresponding to a set of target input channels, switch the adder group corresponding to the multiplier group to the first adder group.

[0166] In this embodiment, the adder group corresponding to the multiplier group may be, but is not limited to, two types, namely the first adder group and the second adder group.

[0167] The first adder group is used to perform addition operations on the multiplication results corresponding to the multiplier group.

[0168] The second adder group is used to perform addition operations on the multiplication results corresponding to the multiplier group and the multiplication results corresponding to other multiplier groups besides the multiplier group.

[0169] Among them, a set of adders can be set up for the corresponding multiplier group. The set of adders includes a first adder section, a second adder section and a third adder section. The output of the first adder section can be used as the input of the second adder section or the third adder section.

[0170] like Figure 13 As shown, the first adder section and the second adder section can form a first adder group, and the first adder section and the third adder section can form a second adder group.

[0171] Specifically, the output of the first adder part can be used as the input of the second adder part according to the first deployed model; or the output of the first adder part can be used as the input of the third adder part according to the second deployed model, so as to realize that a set of adders can run different models at the same time.

[0172] In this embodiment, by reusing the first adder section, the space occupied by the adder group can be saved and the deployment cost of the adder group can be reduced.

[0173] Of course, two sets of adders can also be set up to correspond to the multiplier set, such as Figure 14 As shown, the multiplier group can correspond to the first adder group and the second adder group. The first adder group and the second adder group can be switched according to the deployed model.

[0174] Of course, if two models are deployed at the same time, the two sets of adders can run both models simultaneously.

[0175] Step S33, if the multiplier group supports multiplication operation on the target data corresponding to each of the target input channel groups, the target data corresponding to each of the target input channel groups is allocated to different multiplier groups in the at least one multiplier group, so that the multiplier group performs multiplication operation on the target data corresponding to each of the target input channel groups and the convolution operator in the convolution kernel, to obtain the multiplication operation result corresponding to each of the target input channel groups, so that the first adder group corresponding to the multiplier group performs addition operation on the multiplication operation result corresponding to each of the target input channel groups.

[0176] In this embodiment, the detailed process of step S33 can refer to the related introduction of step S22 in embodiment 4, which will not be repeated here.

[0177] Step S34, if the multiplier group does not support multiplication operation on the target data corresponding to each of the target input channel groups, the adder group corresponding to the multiplier group is switched to the second adder group.

[0178] Step S35, if the multiplier group does not support multiplication operation on the target data corresponding to each of the target input channel groups, a plurality of first target data is obtained from the target data corresponding to each of the target input channel groups based on the size of the multiplier group, to obtain the first target data corresponding to each of the target input channel groups.

[0179] Step S36, the first target data corresponding to each of the target input channel groups is allocated to different first multiplier groups in the at least one multiplier group, so that the first multiplier group performs multiplication operation on the first target data corresponding to each of the target input channel groups and the convolution operator in the convolution kernel, to obtain the first multiplication operation result corresponding to each of the target input channel groups.

[0180] Step S37, the second target data in the target data corresponding to each of the target input channel groups except the first target data is allocated to the second multiplier group in the at least one multiplier group, so that the second multiplier group performs multiplication operation on the second target data corresponding to each of the target input channel groups and the corresponding convolution operator in the convolution kernel, to obtain the second multiplication operation result corresponding to each of the target input channel groups.

[0181] Step S38, the first multiplication operation result corresponding to each of the target input channel groups is added by the second adder group corresponding to the first multiplier group, to obtain the first addition operation result, and the first addition operation result and the second multiplication operation result corresponding to each of the target input channel groups are added, to obtain the second addition operation result.

[0182] For example, as Figure 12As shown, the adder group J1 corresponding to the multiplier group X1 is a second adder group corresponding to the multiplier group X1, and the adder group J1 performs addition operation on the first multiplication result corresponding to the target input channel d1 to obtain a first addition result corresponding to the target input channel d1, performs addition operation on the first multiplication result corresponding to the target input channel d2 to obtain a first addition result corresponding to the target input channel d2, performs addition operation on the first addition result corresponding to the target input channel d1 and the second multiplication result corresponding to the target input channel d1 obtained by the multiplier group X9 to obtain a second addition result corresponding to the target input channel d1, and performs addition operation on the first addition result corresponding to the target input channel d2 and the second multiplication result corresponding to the target input channel d2 obtained by the multiplier group X AK The adder group J1 performs addition operation on the first addition result corresponding to the target input channel d1 and the second multiplication result corresponding to the target input channel d1 obtained by the multiplier group X

[0183] The detailed processes of steps S35-S38 can refer to the related descriptions of steps S23-S26 in Embodiment 4, which will not be repeated here.

[0184] In this embodiment, the adder group corresponding to the multiplier group can be but is not limited to two kinds, and the two kinds of adder groups can be switched for use as needed, so that at least one multiplier group and at least one adder group can be applied to different application scenarios, improving the flexibility of hardware use.

[0185] Next, a data processing apparatus provided by the present application is introduced, and the data processing apparatus introduced below can be correspondingly referred to the data processing method introduced above.

[0186] In this embodiment, the data processing apparatus comprises a buffering unit, an acquisition unit and a distribution unit.

[0187] The buffering unit is configured to buffer data with consistent row attributes in the data of each target input channel of the convolutional neural network into the same memory to obtain data of each target input channel corresponding to at least one row attribute, and the data of each target input channel corresponds to the size of the multiplier group in the at least one multiplier group and the size of the convolution kernel in the plurality of convolution kernels used for separable convolution operation.

[0188] The acquisition unit is configured to acquire at least one target data from the data of each target input channel corresponding to at least one row attribute in parallel based on the plurality of convolution kernels to obtain target data corresponding to each target input channel.

[0189] The distribution unit is configured to distribute target data corresponding to each target input channel to different multiplier groups in at least one multiplier group, so that the multiplier groups multiply the target data corresponding to the target input channel and a convolution operator in a convolution kernel to obtain multiplication results, and so that an adder group corresponding to the multiplier group adds the multiplication results.

[0190] In this embodiment, the data processing apparatus can further include:

[0191] The writing unit is configured to write data of each target input channel corresponding to each row attribute into the shift register matrix.

[0192] Correspondingly, the acquisition unit can be specifically configured to:

[0193] If the size of the convolution kernel supported by the multiplier is not smaller than the size of the convolution kernel, the data of each target input channel corresponding to at least one row attribute is acquired from the shift register matrix based on the plurality of convolution kernels at one time, to obtain each target data of each target input channel.

[0194] In this embodiment, the distribution unit can be specifically configured to:

[0195] Grouping each target input channel to obtain at least one group of target input channels.

[0196] If the multiplier group supports multiplication operation on the target data corresponding to the group of target input channels, the target data corresponding to each group of target input channels in the at least one group of target input channels is distributed to different multiplier groups in at least one multiplier group, so that the multiplier groups multiply the target data corresponding to each target input channel in the group of target input channels and a convolution operator in a convolution kernel to obtain multiplication results corresponding to each target input channel, and so that an adder group corresponding to the multiplier group adds the multiplication results corresponding to each target input channel.

[0197] In this embodiment, the distribution unit can be further configured to:

[0198] If the multiplier group does not support multiplication operation on the target data corresponding to the group of target input channels, a plurality of first target data is obtained from the target data corresponding to each target input channel in each group of target input channels based on the size of the multiplier group, to obtain first target data corresponding to each group of target input channels.

[0199] The first target data corresponding to each group of target input channels is distributed to different first multiplier groups in at least one multiplier group, so that the first multiplier groups multiply the first target data corresponding to the group of target input channels and a convolution operator in a convolution kernel to obtain first multiplication results corresponding to each target input channel.

[0200] The second target data, excluding the first target data, in the target data corresponding to each target input channel is assigned to the second multiplier group in at least one multiplier group, so that the second multiplier group performs multiplication operation on the second target data corresponding to each target input channel and the corresponding convolution operator in the convolution kernel to obtain the second multiplication operation result corresponding to each target input channel;

[0201] The first multiplication result for each target input channel is added to the first multiplication result based on the adder group corresponding to the first multiplier group. The first addition result and the second multiplication result for each target input channel are then added to obtain the second addition result.

[0202] In this embodiment, the data processing device may further include:

[0203] The switching unit is used to switch the adder group corresponding to the multiplier group to the first adder group if the multiplier group supports multiplication of the target data corresponding to a set of target input channels. The first adder group is used to perform addition on the multiplication results corresponding to the multiplier group.

[0204] If the multiplier group does not support multiplication of the target data corresponding to a set of target input channels, the adder group corresponding to the multiplier group is switched to the second adder group. The second adder group is used to add the multiplication results corresponding to the multiplier group and the multiplication results corresponding to other multiplier groups other than the multiplier group.

[0205] In this embodiment, the allocation unit obtains multiple first target data from the target data corresponding to each target input channel in each group of target input channels based on the size of the multiplier group. The process of obtaining the first target data corresponding to each group of target input channels may specifically include:

[0206] Based on the size of the multiplier group, multiple first target data are obtained from the target data corresponding to each target input channel in each target input channel group, and the number of first target data corresponding to each target input channel group is consistent with the number of multipliers in the multiplier group.

[0207] Corresponding to the data processing method embodiment provided in this application above, this application also provides an accelerator embodiment for convolutional neural networks.

[0208] like Figure 15 The diagram shown is a structural schematic of an accelerator for a convolutional neural network provided in this application. The accelerator may include:

[0209] at least one cache memory, at least one multiplier group, and at least one adder group.

[0210] The at least one cache memory can be denoted as cache memories M1, M2, … Mn, n being an integer not less than 1.

[0211] The at least one multiplier group can be denoted as AK multiplier groups, each of which includes AC multipliers, AK being an integer not less than 1, and AC being an integer not less than 1.

[0212] The cache memory is configured to buffer data with consistent row attributes in data of each target input channel of the convolutional neural network, the data of each target input channel corresponding to a size of a multiplier group in the at least one multiplier group and a size of a convolution kernel in a plurality of convolution kernels used for separable convolution operation.

[0213] The multiplier group is configured to perform multiplication operation on target data corresponding to a target input channel obtained by distribution and a convolution operator in the convolution kernel to obtain a multiplication operation result, the target data corresponding to the target input channel being obtained based on the plurality of convolution kernels and at least one target data obtained from data of each target input channel corresponding to each row attribute in parallel.

[0214] The adder group is configured to perform addition operation on multiplication operation results of the multiplier group corresponding to the adder group.

[0215] In the embodiment, the adder group can include a first adder part, a second adder part, and a third adder part, and an output of the first adder part can be used as an input of the second adder part or the third adder part.

[0216] The first adder part and the second adder part can constitute a first adder group, and the first adder part and the third adder part can constitute a second adder group.

[0217] Of course, the adder group can also include the first adder group and the second adder group.

[0218] The first adder group is configured to perform addition operation on multiplication operation results of the multiplier group.

[0219] The second adder group is configured to perform addition operation on multiplication operation results of the multiplier group and multiplication operation results of other multiplier groups except the multiplier group.

[0220] The number of the at least one memory can be consistent with the number of one column of weight data of the convolution kernel.

[0221] The capacity of the memory can be not less than a data amount corresponding to data with consistent row attributes in data of each target input channel of the convolutional neural network.

[0222] The accelerator can further include a shift register matrix configured to store data of each target input channel corresponding to each row attribute.

[0223] It should be noted that each of the arrangements described with respect to the various embodiments focuses on the differences from the other arrangements, and that the same or similar parts between the arrangements are cross-referenced. For the device embodiments, the description is relatively simple as the device embodiments are substantially similar to the method embodiments, and the relevant parts are cross-referenced to the description of the method embodiments.

[0224] Finally, it should be noted that the terms such as first and second, etc., are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by an "including a" statement does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0225] For the convenience of description, the above device is described in various modules according to functions. Of course, the functions of the modules can be implemented in the same or more software and / or hardware when implementing the present application.

[0226] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0227] The above describes in detail the data processing method, device, and accelerator of the convolutional neural network provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed; and in summary, the content of the specification should not be understood as limiting the present application.

Claims

1. A data processing method, comprising: Data with consistent row attributes in each target input channel of the convolutional neural network are buffered into the same memory to obtain data for each target input channel corresponding to the row attributes. The data of each target input channel corresponds to the size of the multiplier group and the size of the convolution kernel in the multiple convolution kernels used for separable convolution operations. Based on the multiple convolution kernels, the target data corresponding to each target input channel is obtained in parallel from the data of each target input channel corresponding to each row attribute; The target data corresponding to each target input channel is assigned to different multiplier groups, so that the multiplier groups perform multiplication operations on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain the multiplication operation result, so that the adder group corresponding to the multiplier group performs addition operation on the multiplication operation result.

2. The method according to claim 1, further comprising: Write the data of each target input channel corresponding to each row attribute into the shift register matrix; The step of obtaining target data corresponding to each target input channel from the data of each target input channel corresponding to each row attribute in parallel based on the multiple convolutional kernels includes: If the size of the convolution kernel supported by the multiplier is not less than the size of the convolution kernel, the data of each target input channel corresponding to the row attribute is obtained from the shift register matrix at one time based on the multiple convolution kernels, so as to obtain the target data of each target input channel.

3. The method according to claim 1 or 2, wherein the step of allocating the target data corresponding to each target input channel to different multiplier groups, so that the multiplier groups perform multiplication operations on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain a multiplication result, and the adder group corresponding to the multiplier group performs addition operations on the multiplication result, includes: The target input channels are grouped to obtain at least one group of target input channels; If the multiplier group supports multiplication of the target data corresponding to the set of target input channels, the target data corresponding to each of the at least one set of target input channels is assigned to different multiplier groups, so that the multiplier group performs multiplication of the target data corresponding to each of the target input channels in the set of target input channels with the convolution operator in the convolution kernel to obtain the multiplication result corresponding to each target input channel, so that the adder group corresponding to the multiplier group performs addition of the multiplication result corresponding to each target input channel.

4. The method according to claim 3, further comprising: If the multiplier group does not support multiplication of the target data corresponding to the group of target input channels, based on the size of the multiplier group, multiple first target data are obtained from the target data corresponding to each target input channel in each group of target input channels, thus obtaining the first target data corresponding to each group of target input channels; The first target data corresponding to each group of target input channels is assigned to different first multiplier groups, so that the first multiplier groups perform multiplication operations on the first target data corresponding to each group of target input channels and the convolution operator in the convolution kernel to obtain the first multiplication operation result corresponding to each target input channel; The second target data, excluding the first target data, in the target data corresponding to each target input channel is allocated to the second multiplier group, so that the second multiplier group performs multiplication operations on the second target data corresponding to each target input channel and the corresponding convolution operator in the convolution kernel to obtain the second multiplication operation result corresponding to each target input channel; Based on the adder group corresponding to the first multiplier group, the first multiplication result corresponding to each target input channel is added to obtain the first addition result. The first addition result and the second multiplication result corresponding to each target input channel are then added to obtain the second addition result.

5. The method according to claim 4, further comprising: If the multiplier group supports multiplication of the target data corresponding to the set of target input channels, the adder group corresponding to the multiplier group is switched to the first adder group, which is used to perform addition on the multiplication result corresponding to the multiplier group. If the multiplier group does not support multiplication of the target data corresponding to the set of target input channels, the adder group corresponding to the multiplier group is switched to a second adder group. The second adder group is used to perform addition operations on the multiplication results corresponding to the multiplier group and the multiplication results corresponding to other multiplier groups other than the multiplier group.

6. The method according to claim 4, wherein obtaining multiple first target data from the target data corresponding to each target input channel in each group of target input channels based on the size of the multiplier group, thereby obtaining the first target data corresponding to each group of target input channels, comprises: Based on the size of the multiplier group, multiple first target data are obtained from the target data corresponding to each target input channel in each group of target input channels, resulting in the first target data corresponding to each group of target input channels. The number of first target data corresponding to each group of target input channels is consistent with the number of multipliers in the multiplier group.

7. A data processing apparatus, comprising: A buffer unit is used to buffer data with consistent row attributes from each target input channel of the convolutional neural network into the same memory to obtain the data of each target input channel corresponding to the row attributes. The data of each target input channel corresponds to the size of the multiplier group and the size of the convolution kernel in the multiple convolution kernels used for separable convolution operations. The acquisition unit is used to acquire target data corresponding to each target input channel from the data of each target input channel corresponding to each row attribute in parallel based on the plurality of convolution kernels; The allocation unit is used to allocate the target data corresponding to each target input channel to different multiplier groups, so that the multiplier groups perform multiplication operations on the target data corresponding to the target input channel and the convolution operator in the convolution kernel to obtain the multiplication operation result, so that the adder group corresponding to the multiplier group performs addition operation on the multiplication operation result.

8. An accelerator for a convolutional neural network, comprising: Memory, multiplier set, and adder set; The memory is used to buffer data with consistent row attributes in each target input channel of the convolutional neural network, and the data in each target input channel corresponds to the size of the multiplier group and the size of the convolution kernel in the plurality of convolution kernels used for separable convolution operations; The target data corresponding to each target input channel is assigned to different multiplier groups, so that the multiplier groups perform multiplication operations on the target data corresponding to the assigned target input channel and the convolution operator in the convolution kernel to obtain the multiplication operation result. The target data corresponding to the target input channel is obtained in parallel from the data of each target input channel corresponding to each row attribute based on the multiple convolution kernels. The adder group is used to perform addition operations on the multiplication results of the multiplier group corresponding to the adder group.

9. The accelerator of claim 8, wherein the adder assembly comprises: At least one first adder group and at least one second adder group; The first adder group is used to perform addition operations on the multiplication results corresponding to the multiplier group; The second adder group is used to perform addition operations on the multiplication results corresponding to the multiplier group and the multiplication results corresponding to other multiplier groups besides the multiplier group.

10. The accelerator according to claim 8, wherein the number of memories is the same as the number of weight data in a column of the convolution kernel; The capacity of the memory is not less than the amount of data corresponding to data with consistent row attributes in each target input channel of the convolutional neural network.

11. The accelerator of claim 8, further comprising: A shift register matrix, wherein the shift register matrix is ​​used to store the data of each target input channel corresponding to each row attribute.

Images